Basic Web Crawler

Abstract

A web crawler is a program that visits web pages, reads them, follows the links it finds, and repeats. Search engines crawl the web at massive scale; you can build a tiny version in under a hundred lines of Python. In this project you will build a polite, breadth-first crawler that starts at a URL, stays within the same domain, extracts structured data from each page (title, meta description, headings), and saves the results to a CSV file.

The goals are real:

Understand how HTTP, HTML parsing, and link discovery fit together.
Use Python’s requests library to fetch pages.
Use BeautifulSoup to parse HTML.
Manage a frontier (the queue of URLs to visit) and a visited set (so we never visit the same page twice).
Crawl politely — with delays, domain restrictions, and respect for robots.txt.

Crawling 101

The simplest crawler is just three loops:

Take the next URL from the frontier.
Fetch and parse it.
Add every link you find back to the frontier (if you have not seen it).

What separates a useful crawler from a server-crashing pest:

Limit the rate of requests so you do not hammer the site.
Stay on-domain unless you explicitly want to crawl the wider web.
Respect robots.txt — the file at /robots.txt tells crawlers which paths are off-limits.
Identify yourself with a clear User-Agent header that includes contact info.
Bound your work — set a max page count so you do not crawl forever.

Prerequisites

Python 3.6 or above.
A text editor or IDE.
Internet connection.
Basic understanding of HTML structure (tags, attributes).
Familiarity with functions and classes.

Install Dependencies

install

pip install requests beautifulsoup4

diagram how the pieces call each other mermaid

  flowchart TD
  n0(["script start"])
  n2["main()"]
  subgraph WebCrawler
    n45["crawl()"]
    n46["extract_links()"]
    n47["extract_page_data()"]
    n48["get_page_content()"]
    n49["is_valid_url()"]
    n50["print_summary()"]
    n51["save_to_csv()"]
  end
  n45 --> n46
  n45 --> n47
  n45 --> n48
  n46 --> n49
  n47 --> n46
  n0 --> n2
  n2 --> n45
  n2 --> n50
  n2 --> n51

Derived from projects/beginners/basicwebcrawler.py by parsing it, not by hand. Arrows are calls between the file's own functions and methods; library calls are left out, and only calls the parser could resolve with certainty are shown.

Getting Started

Create the project

Create a folder named basic-web-crawler.
Inside it, create basicwebcrawler.py.

Write the code

Basic Web Crawler

pch.viewSource

Basic Web Crawler

# Basic Web Crawler

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse
import time
import csv
from collections import deque


def ask(prompt="", default=""):
    """Read a line, or fall back to `default` when nobody is there to type.

    Without this the script raises EOFError the moment it runs unattended — in
    a test, a scheduled job, or the build that captures this output for the
    docs. The fallback is printed rather than silent, so a reader can always
    tell which answers were typed and which were assumed.
    """
    try:
        return input(prompt).strip() or default
    except EOFError:
        print(f"{default}   (no input available, using the default)")
        return default

class WebCrawler:
    def __init__(self, start_url, max_pages=10, delay=1):
        self.start_url = start_url
        self.max_pages = max_pages
        self.delay = delay
        self.visited_urls = set()
        self.to_visit = deque([start_url])
        self.crawled_data = []
        
    def is_valid_url(self, url):
        """Check if URL is valid and belongs to the same domain"""
        try:
            parsed = urlparse(url)
            return bool(parsed.netloc) and bool(parsed.scheme)
        except:
            return False
    
    def get_page_content(self, url):
        """Fetch and parse page content"""
        try:
            headers = {
                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
            }
            response = requests.get(url, headers=headers, timeout=10)
            response.raise_for_status()
            return response.text
        except requests.RequestException as e:
            print(f"Error fetching {url}: {e}")
            return None
    
    def extract_links(self, html, base_url):
        """Extract all links from HTML content"""
        soup = BeautifulSoup(html, 'html.parser')
        links = []
        
        for link in soup.find_all('a', href=True):
            href = link['href']
            full_url = urljoin(base_url, href)
            
            if self.is_valid_url(full_url):
                links.append(full_url)
        
        return links
    
    def extract_page_data(self, html, url):
        """Extract useful data from the page"""
        soup = BeautifulSoup(html, 'html.parser')
        
        # Extract title
        title = soup.find('title')
        title_text = title.get_text().strip() if title else "No Title"
        
        # Extract meta description
        meta_desc = soup.find('meta', attrs={'name': 'description'})
        description = meta_desc.get('content', '') if meta_desc else ''
        
        # Extract headings
        headings = []
        for heading in soup.find_all(['h1', 'h2', 'h3']):
            headings.append(heading.get_text().strip())
        
        # Extract text content (first 200 chars)
        text_content = soup.get_text()
        clean_text = ' '.join(text_content.split())[:200] + '...'
        
        return {
            'url': url,
            'title': title_text,
            'description': description,
            'headings': headings[:5],  # First 5 headings
            'content_preview': clean_text,
            'links_count': len(self.extract_links(html, url))
        }
    
    def crawl(self):
        """Main crawling function"""
        print(f"Starting crawl from: {self.start_url}")
        print(f"Max pages: {self.max_pages}")
        print("-" * 50)
        
        pages_crawled = 0
        
        while self.to_visit and pages_crawled < self.max_pages:
            current_url = self.to_visit.popleft()
            
            if current_url in self.visited_urls:
                continue
            
            print(f"Crawling: {current_url}")
            
            # Fetch page content
            html = self.get_page_content(current_url)
            if html is None:
                continue
            
            # Mark as visited
            self.visited_urls.add(current_url)
            
            # Extract page data
            page_data = self.extract_page_data(html, current_url)
            self.crawled_data.append(page_data)
            
            print(f"  Title: {page_data['title']}")
            print(f"  Links found: {page_data['links_count']}")
            
            # Extract and queue new links
            links = self.extract_links(html, current_url)
            for link in links:
                if link not in self.visited_urls:
                    # Only crawl within the same domain
                    if urlparse(link).netloc == urlparse(self.start_url).netloc:
                        self.to_visit.append(link)
            
            pages_crawled += 1
            
            # Be respectful - add delay
            time.sleep(self.delay)
        
        print(f"\nCrawling completed! Visited {pages_crawled} pages.")
        return self.crawled_data
    
    def save_to_csv(self, filename="crawl_results.csv"):
        """Save crawled data to CSV file"""
        if not self.crawled_data:
            print("No data to save.")
            return
        
        with open(filename, 'w', newline='', encoding='utf-8') as csvfile:
            fieldnames = ['url', 'title', 'description', 'headings', 'content_preview', 'links_count']
            writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
            
            writer.writeheader()
            for data in self.crawled_data:
                # Convert headings list to string
                data_copy = data.copy()
                data_copy['headings'] = '; '.join(data['headings'])
                writer.writerow(data_copy)
        
        print(f"Results saved to {filename}")
    
    def print_summary(self):
        """Print crawling summary"""
        if not self.crawled_data:
            print("No data crawled.")
            return
        
        print(f"\nCRAWL SUMMARY")
        print("=" * 50)
        print(f"Total pages crawled: {len(self.crawled_data)}")
        print(f"Total unique URLs visited: {len(self.visited_urls)}")
        
        print(f"\nPages found:")
        for i, data in enumerate(self.crawled_data, 1):
            print(f"{i:2d}. {data['title'][:50]}...")
            print(f"     {data['url']}")

def main():
    # Example usage
    start_url = ask("Enter the starting URL to crawl: ", 'https://example.com').strip()
    if not start_url:
        start_url = "https://example.com"
    
    try:
        max_pages = int(ask("Enter maximum pages to crawl (default 5): ", '7') or "5")
    except ValueError:
        max_pages = 5
    
    print(f"\nStarting web crawler...")
    crawler = WebCrawler(start_url, max_pages=max_pages, delay=1)
    
    try:
        crawled_data = crawler.crawl()
        crawler.print_summary()
        
        save_choice = ask("\nSave results to CSV? (y/n): ", 'n').lower()
        if save_choice == 'y':
            crawler.save_to_csv()
    
    except KeyboardInterrupt:
        print("\nCrawling interrupted by user.")
    except Exception as e:
        print(f"Error during crawling: {e}")

if __name__ == "__main__":
    main()

Run it

run

python basicwebcrawler.py

The crawler prompts for a start URL, max page count, and whether to save a CSV. Sample run:

text

Start URL: https://example.com
Max pages: 5
Save CSV? (y/n): y
[1/5] Fetched https://example.com (200) — 3 new links
[2/5] Fetched https://example.com/about (200) — 1 new link
...
Saved 5 rows to crawl_results.csv

What it produces

Running the file exactly as it ships takes 2.1 s and prints:

python basicwebcrawler.py

Enter the starting URL to crawl: https://example.com   (no input available, using the default)
Enter maximum pages to crawl (default 5): 7   (no input available, using the default)
 
Starting web crawler...
Starting crawl from: https://example.com
Max pages: 7
--------------------------------------------------
Crawling: https://example.com
  Title: Example Domain
  Links found: 1
 
Crawling completed! Visited 1 pages.
 
CRAWL SUMMARY
==================================================
Total pages crawled: 1
Total unique URLs visited: 1
 
Pages found:
 1. Example Domain...
...

The first 20 of 23 lines are shown; the run continues past this point.

Step-by-Step Explanation

1. Imports and class skeleton

basicwebcrawler.py

from collections import deque
from urllib.parse import urljoin, urlparse
import csv
import time
import requests
from bs4 import BeautifulSoup
 
class WebCrawler:
    def __init__(self, start_url, max_pages=10, delay=1):
        self.start_url = start_url
        self.max_pages = max_pages
        self.delay = delay
        self.visited_urls = set()
        self.to_visit = deque([start_url])
        self.crawled_data = []

deque (double-ended queue) gives O(1) popleft() and append(), which is exactly what a FIFO frontier needs.
visited_urls is a set so duplicate detection is O(1).
crawled_data accumulates one dict per page for CSV export.

2. URL validation

basicwebcrawler.py

def is_valid_url(self, url):
    parsed = urlparse(url)
    return bool(parsed.netloc) and bool(parsed.scheme)

A valid URL has both a scheme (http, https) and a netloc (example.com). Without either, it cannot be fetched.

3. Extract links

basicwebcrawler.py

def extract_links(self, html, base_url):
    soup = BeautifulSoup(html, "html.parser")
    links = []
    for a in soup.find_all("a", href=True):
        absolute = urljoin(base_url, a["href"])
        if self.is_valid_url(absolute):
            links.append(absolute)
    return links

urljoin(base, href) turns relative links like /about into absolute URLs like https://example.com/about. Always do this — relative paths are useless to requests.get.

4. Extract page data

basicwebcrawler.py

def extract_page_data(self, html, url):
    soup = BeautifulSoup(html, "html.parser")
    title = soup.title.get_text(strip=True) if soup.title else ""
    meta = soup.find("meta", attrs={"name": "description"})
    description = meta["content"] if meta and meta.get("content") else ""
    headings = [h.get_text(strip=True) for h in soup.find_all(["h1", "h2", "h3"])]
    return {
        "url": url,
        "title": title,
        "description": description,
        "headings": " | ".join(headings)[:500],
    }

This grabs the bits most analysts care about: title, meta description, headings.

5. The main loop

basicwebcrawler.py

def crawl(self):
    while self.to_visit and len(self.visited_urls) < self.max_pages:
        url = self.to_visit.popleft()
        if url in self.visited_urls:
            continue
        try:
            response = requests.get(url, timeout=10, headers={
                "User-Agent": "BasicCrawler/1.0 (+contact@example.com)"
            })
            response.raise_for_status()
        except requests.RequestException as e:
            print(f"Failed {url}: {e}")
            continue
 
        self.visited_urls.add(url)
        self.crawled_data.append(self.extract_page_data(response.text, url))
 
        for link in self.extract_links(response.text, url):
            if urlparse(link).netloc == urlparse(self.start_url).netloc:
                if link not in self.visited_urls:
                    self.to_visit.append(link)
 
        time.sleep(self.delay)

Key points:

Timeout keeps a hanging server from freezing the crawl.
raise_for_status() turns 4xx/5xx responses into exceptions.
Domain check keeps us from accidentally crawling Wikipedia from example.com.
time.sleep(self.delay) is the polite delay between requests.

6. CSV export

basicwebcrawler.py

def save_to_csv(self, filename="crawl_results.csv"):
    if not self.crawled_data:
        return
    fieldnames = self.crawled_data[0].keys()
    with open(filename, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=fieldnames)
        writer.writeheader()
        writer.writerows(self.crawled_data)

csv.DictWriter matches dictionary keys to columns automatically.

Respecting `robots.txt`

Politeness is not optional. Use Python’s built-in urllib.robotparser:

robots.py

from urllib.robotparser import RobotFileParser
 
rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
 
if rp.can_fetch("BasicCrawler/1.0", url):
    response = requests.get(url, ...)
else:
    print(f"Skipping {url} — disallowed by robots.txt")

Most public sites set their crawler rules in /robots.txt. Ignoring those rules is rude and can get your IP banned.

Common Mistakes

Problem	Cause	Fix
Crawler runs forever	No max-pages limit	Always cap `len(visited_urls) < max_pages`
403 Forbidden everywhere	Default `requests` User-Agent	Set a descriptive `User-Agent` header
Same page visited many times	Trailing slash or query string differences	Normalize URLs before adding to the visited set
Memory grows unbounded	Storing every page’s HTML	Store only the extracted fields
`RecursionError`	Recursive crawl instead of iterative	Use the `deque`-based loop above
Server returns 429	Hitting too fast	Increase the delay or back off on 429

Variations to Try

1. Depth-first vs. breadth-first

Swap popleft() for pop() and the queue becomes a stack — depth-first crawling. Useful for very narrow, deep sites.

2. Concurrent crawling

Use concurrent.futures.ThreadPoolExecutor to fetch several URLs in parallel — but rate-limit per domain. Anything else is impolite.

3. Sitemap-aware crawling

Fetch /sitemap.xml first to seed the frontier with URLs the site wants indexed.

4. Async crawler

Rewrite with aiohttp + asyncio for higher throughput at the same politeness budget.

5. Download files

Extend the crawler to grab .pdf, .jpg, etc. Save under a folder named after the domain.

6. Headless browser

Some sites render content with JavaScript and serve empty HTML to requests. Use Playwright or Selenium to render the page first.

7. Persistent state

Save visited_urls and to_visit to a JSON file every minute so you can resume after a crash.

8. Incremental indexing

After each fetch, push the page text into a SQLite full-text index. You now have a personal Google.

Crawling Ethics & Legality

Crawling is powerful and risky. Before you point your crawler at a real site:

Read its Terms of Service. Many sites explicitly forbid automated access.
Honor robots.txt.
Avoid crawling content behind login forms unless you have permission.
Set a polite delay — at least 1 second between requests by default.
Identify your crawler with a real User-Agent and contact URL.
Stop immediately if the site asks you to.

Hammering a small site can cost the owner real money in bandwidth bills. Treat every fetch like a tiny favor someone is doing you.

Real-World Applications

Search engines — Google’s crawler is the most famous example.
Price-tracking bots — comparing items across e-commerce sites (with permission).
SEO audits — building a sitemap, finding broken internal links.
News aggregators — collecting recent headlines for analysis.
Academic research — gathering corpora for NLP training (where licenses allow).

Educational Value

This project teaches:

HTTP — requests, status codes, headers.
HTML parsing with BeautifulSoup.
URL manipulation with urllib.parse.
Data structures — queues for the frontier, sets for the visited list.
OOP — wrapping state and behavior in a class.
Ethics — politeness is engineering, not a nice-to-have.

Next Steps

Combine with RSS Feed Reader to crawl articles automatically.
Add content classification with a small ML model (scikit-learn).
Build a search interface with Flask (see Basic Web Server).
Deploy as a scheduled job — re-crawl every day, diff the results, alert on changes.
Move to a real crawler framework like Scrapy when your needs outgrow a single script.

Conclusion

You wrote a real, working web crawler — not a toy. It fetches pages politely, parses them, follows links, stays on-domain, and exports structured results. The patterns scale: a thousand-line Scrapy spider is still doing the same three things in a loop. Full source on GitHub. Explore more web projects on Python Central Hub.

Pitfalls

max_pages is a ceiling, not a promise. The run above asked for 7 pages and visited 1. example.com has exactly one link, it points at iana.org, and the same-domain rule drops it — so the frontier emptied on the first iteration. A crawler that reports “crawled 1 page” has usually hit this, not a bug.
except: with no exception type. is_valid_url uses a bare except, which swallows KeyboardInterrupt and SystemExit along with the parse error it means to catch. except ValueError: is the fix, and it is one word.
No seen set means no termination. Pages link back to their own root. With the visited set removed, the exercise below still has 5 URLs queued after 30 steps and would keep going indefinitely.
time.sleep(delay) is the politeness, and it is easy to delete. One second per page is what stops this from being a load test against someone else’s server. It is also why the crawl took 2.5 s for a single page.
robots.txt is not consulted anywhere in this file. Reading it is urllib.robotparser and about four lines; shipping a crawler without it is a decision, not an oversight.

Recap

Measured run: 7 pages requested, 1 visited, 2.5 s, one link found and rejected as off-domain.
The frontier is a deque; popleft() makes it breadth-first, pop() makes it depth-first, and that single call is the entire difference.
The visited set is what makes the crawl finite, not the page limit.
Same-domain filtering is what stops a crawl of one site becoming a crawl of the web.

pch.quizTag pch.quizDefaultTitle

The crawler was told to visit up to 7 pages and visited 1. Why?

pch.quizShowAnswer

B — example.com has a single outbound link, to another domain, which the same-domain filter rejects — so the frontier was empty after the first page
What changes if deque.popleft() is replaced with deque.pop()?

pch.quizShowAnswer

B — The frontier becomes a stack instead of a queue, turning breadth-first crawling into depth-first — the same pages are reached, in a different order
Which piece of this crawler is what guarantees it terminates?

pch.quizShowAnswer

B — The visited set — without it, pages that link back to their root are re-queued forever, and the page limit only hides that

Try it yourself

pch.coffeeTagline

pch.coffeeCta

pch.feedbackHeading

pch.feedbackSubheading

Basic Web Crawler

Abstract

Crawling 101

Prerequisites

Install Dependencies

Getting Started

Create the project

Write the code

Run it

What it produces

Step-by-Step Explanation

1. Imports and class skeleton

2. URL validation

3. Extract links

4. Extract page data

5. The main loop

6. CSV export

Respecting robots.txt

Common Mistakes

Variations to Try

1. Depth-first vs. breadth-first

2. Concurrent crawling

3. Sitemap-aware crawling

4. Async crawler

5. Download files

6. Headless browser

7. Persistent state

8. Incremental indexing

Crawling Ethics & Legality

Real-World Applications

Educational Value

Next Steps

Conclusion

Pitfalls

Recap

Try it yourself

pch.feedbackHeading

Respecting `robots.txt`