Skip to content
Python Central Hub

Basic Web Crawler

View on GitHub

Abstract

A web crawler is a program that visits web pages, reads them, follows the links it finds, and repeats. Search engines crawl the web at massive scale; you can build a tiny version in under a hundred lines of Python. In this project you will build a polite, breadth-first crawler that starts at a URL, stays within the same domain, extracts structured data from each page (title, meta description, headings), and saves the results to a CSV file.

The goals are real:

  • Understand how HTTP, HTML parsing, and link discovery fit together.
  • Use Python’s requestsrequests library to fetch pages.
  • Use BeautifulSoup to parse HTML.
  • Manage a frontier (the queue of URLs to visit) and a visited set (so we never visit the same page twice).
  • Crawl politely — with delays, domain restrictions, and respect for robots.txtrobots.txt.

Crawling 101

The simplest crawler is just three loops:

  1. Take the next URL from the frontier.
  2. Fetch and parse it.
  3. Add every link you find back to the frontier (if you have not seen it).

What separates a useful crawler from a server-crashing pest:

  • Limit the rate of requests so you do not hammer the site.
  • Stay on-domain unless you explicitly want to crawl the wider web.
  • Respect robots.txtrobots.txt — the file at /robots.txt/robots.txt tells crawlers which paths are off-limits.
  • Identify yourself with a clear User-AgentUser-Agent header that includes contact info.
  • Bound your work — set a max page count so you do not crawl forever.

Prerequisites

  • Python 3.6 or above.
  • A text editor or IDE.
  • Internet connection.
  • Basic understanding of HTML structure (tags, attributes).
  • Familiarity with functions and classes.

Install Dependencies

install
pip install requests beautifulsoup4
install
pip install requests beautifulsoup4

Getting Started

Create the project

  1. Create a folder named basic-web-crawlerbasic-web-crawler.
  2. Inside it, create basicwebcrawler.pybasicwebcrawler.py.

Write the code

Basic Web CrawlerSource
Basic Web Crawler
# Basic Web Crawler
 
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse
import time
import csv
from collections import deque
 
class WebCrawler:
    def __init__(self, start_url, max_pages=10, delay=1):
        self.start_url = start_url
        self.max_pages = max_pages
        self.delay = delay
        self.visited_urls = set()
        self.to_visit = deque([start_url])
        self.crawled_data = []
        
    def is_valid_url(self, url):
        """Check if URL is valid and belongs to the same domain"""
        try:
            parsed = urlparse(url)
            return bool(parsed.netloc) and bool(parsed.scheme)
        except:
            return False
    
    def get_page_content(self, url):
        """Fetch and parse page content"""
        try:
            headers = {
                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
            }
            response = requests.get(url, headers=headers, timeout=10)
            response.raise_for_status()
            return response.text
        except requests.RequestException as e:
            print(f"Error fetching {url}: {e}")
            return None
    
    def extract_links(self, html, base_url):
        """Extract all links from HTML content"""
        soup = BeautifulSoup(html, 'html.parser')
        links = []
        
        for link in soup.find_all('a', href=True):
            href = link['href']
            full_url = urljoin(base_url, href)
            
            if self.is_valid_url(full_url):
                links.append(full_url)
        
        return links
    
    def extract_page_data(self, html, url):
        """Extract useful data from the page"""
        soup = BeautifulSoup(html, 'html.parser')
        
        # Extract title
        title = soup.find('title')
        title_text = title.get_text().strip() if title else "No Title"
        
        # Extract meta description
        meta_desc = soup.find('meta', attrs={'name': 'description'})
        description = meta_desc.get('content', '') if meta_desc else ''
        
        # Extract headings
        headings = []
        for heading in soup.find_all(['h1', 'h2', 'h3']):
            headings.append(heading.get_text().strip())
        
        # Extract text content (first 200 chars)
        text_content = soup.get_text()
        clean_text = ' '.join(text_content.split())[:200] + '...'
        
        return {
            'url': url,
            'title': title_text,
            'description': description,
            'headings': headings[:5],  # First 5 headings
            'content_preview': clean_text,
            'links_count': len(self.extract_links(html, url))
        }
    
    def crawl(self):
        """Main crawling function"""
        print(f"Starting crawl from: {self.start_url}")
        print(f"Max pages: {self.max_pages}")
        print("-" * 50)
        
        pages_crawled = 0
        
        while self.to_visit and pages_crawled < self.max_pages:
            current_url = self.to_visit.popleft()
            
            if current_url in self.visited_urls:
                continue
            
            print(f"Crawling: {current_url}")
            
            # Fetch page content
            html = self.get_page_content(current_url)
            if html is None:
                continue
            
            # Mark as visited
            self.visited_urls.add(current_url)
            
            # Extract page data
            page_data = self.extract_page_data(html, current_url)
            self.crawled_data.append(page_data)
            
            print(f"  Title: {page_data['title']}")
            print(f"  Links found: {page_data['links_count']}")
            
            # Extract and queue new links
            links = self.extract_links(html, current_url)
            for link in links:
                if link not in self.visited_urls:
                    # Only crawl within the same domain
                    if urlparse(link).netloc == urlparse(self.start_url).netloc:
                        self.to_visit.append(link)
            
            pages_crawled += 1
            
            # Be respectful - add delay
            time.sleep(self.delay)
        
        print(f"\nCrawling completed! Visited {pages_crawled} pages.")
        return self.crawled_data
    
    def save_to_csv(self, filename="crawl_results.csv"):
        """Save crawled data to CSV file"""
        if not self.crawled_data:
            print("No data to save.")
            return
        
        with open(filename, 'w', newline='', encoding='utf-8') as csvfile:
            fieldnames = ['url', 'title', 'description', 'headings', 'content_preview', 'links_count']
            writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
            
            writer.writeheader()
            for data in self.crawled_data:
                # Convert headings list to string
                data_copy = data.copy()
                data_copy['headings'] = '; '.join(data['headings'])
                writer.writerow(data_copy)
        
        print(f"Results saved to {filename}")
    
    def print_summary(self):
        """Print crawling summary"""
        if not self.crawled_data:
            print("No data crawled.")
            return
        
        print(f"\nCRAWL SUMMARY")
        print("=" * 50)
        print(f"Total pages crawled: {len(self.crawled_data)}")
        print(f"Total unique URLs visited: {len(self.visited_urls)}")
        
        print(f"\nPages found:")
        for i, data in enumerate(self.crawled_data, 1):
            print(f"{i:2d}. {data['title'][:50]}...")
            print(f"     {data['url']}")
 
def main():
    # Example usage
    start_url = input("Enter the starting URL to crawl: ").strip()
    if not start_url:
        start_url = "https://example.com"
    
    try:
        max_pages = int(input("Enter maximum pages to crawl (default 5): ") or "5")
    except ValueError:
        max_pages = 5
    
    print(f"\nStarting web crawler...")
    crawler = WebCrawler(start_url, max_pages=max_pages, delay=1)
    
    try:
        crawled_data = crawler.crawl()
        crawler.print_summary()
        
        save_choice = input("\nSave results to CSV? (y/n): ").lower()
        if save_choice == 'y':
            crawler.save_to_csv()
    
    except KeyboardInterrupt:
        print("\nCrawling interrupted by user.")
    except Exception as e:
        print(f"Error during crawling: {e}")
 
if __name__ == "__main__":
    main()
Basic Web Crawler
# Basic Web Crawler
 
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse
import time
import csv
from collections import deque
 
class WebCrawler:
    def __init__(self, start_url, max_pages=10, delay=1):
        self.start_url = start_url
        self.max_pages = max_pages
        self.delay = delay
        self.visited_urls = set()
        self.to_visit = deque([start_url])
        self.crawled_data = []
        
    def is_valid_url(self, url):
        """Check if URL is valid and belongs to the same domain"""
        try:
            parsed = urlparse(url)
            return bool(parsed.netloc) and bool(parsed.scheme)
        except:
            return False
    
    def get_page_content(self, url):
        """Fetch and parse page content"""
        try:
            headers = {
                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
            }
            response = requests.get(url, headers=headers, timeout=10)
            response.raise_for_status()
            return response.text
        except requests.RequestException as e:
            print(f"Error fetching {url}: {e}")
            return None
    
    def extract_links(self, html, base_url):
        """Extract all links from HTML content"""
        soup = BeautifulSoup(html, 'html.parser')
        links = []
        
        for link in soup.find_all('a', href=True):
            href = link['href']
            full_url = urljoin(base_url, href)
            
            if self.is_valid_url(full_url):
                links.append(full_url)
        
        return links
    
    def extract_page_data(self, html, url):
        """Extract useful data from the page"""
        soup = BeautifulSoup(html, 'html.parser')
        
        # Extract title
        title = soup.find('title')
        title_text = title.get_text().strip() if title else "No Title"
        
        # Extract meta description
        meta_desc = soup.find('meta', attrs={'name': 'description'})
        description = meta_desc.get('content', '') if meta_desc else ''
        
        # Extract headings
        headings = []
        for heading in soup.find_all(['h1', 'h2', 'h3']):
            headings.append(heading.get_text().strip())
        
        # Extract text content (first 200 chars)
        text_content = soup.get_text()
        clean_text = ' '.join(text_content.split())[:200] + '...'
        
        return {
            'url': url,
            'title': title_text,
            'description': description,
            'headings': headings[:5],  # First 5 headings
            'content_preview': clean_text,
            'links_count': len(self.extract_links(html, url))
        }
    
    def crawl(self):
        """Main crawling function"""
        print(f"Starting crawl from: {self.start_url}")
        print(f"Max pages: {self.max_pages}")
        print("-" * 50)
        
        pages_crawled = 0
        
        while self.to_visit and pages_crawled < self.max_pages:
            current_url = self.to_visit.popleft()
            
            if current_url in self.visited_urls:
                continue
            
            print(f"Crawling: {current_url}")
            
            # Fetch page content
            html = self.get_page_content(current_url)
            if html is None:
                continue
            
            # Mark as visited
            self.visited_urls.add(current_url)
            
            # Extract page data
            page_data = self.extract_page_data(html, current_url)
            self.crawled_data.append(page_data)
            
            print(f"  Title: {page_data['title']}")
            print(f"  Links found: {page_data['links_count']}")
            
            # Extract and queue new links
            links = self.extract_links(html, current_url)
            for link in links:
                if link not in self.visited_urls:
                    # Only crawl within the same domain
                    if urlparse(link).netloc == urlparse(self.start_url).netloc:
                        self.to_visit.append(link)
            
            pages_crawled += 1
            
            # Be respectful - add delay
            time.sleep(self.delay)
        
        print(f"\nCrawling completed! Visited {pages_crawled} pages.")
        return self.crawled_data
    
    def save_to_csv(self, filename="crawl_results.csv"):
        """Save crawled data to CSV file"""
        if not self.crawled_data:
            print("No data to save.")
            return
        
        with open(filename, 'w', newline='', encoding='utf-8') as csvfile:
            fieldnames = ['url', 'title', 'description', 'headings', 'content_preview', 'links_count']
            writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
            
            writer.writeheader()
            for data in self.crawled_data:
                # Convert headings list to string
                data_copy = data.copy()
                data_copy['headings'] = '; '.join(data['headings'])
                writer.writerow(data_copy)
        
        print(f"Results saved to {filename}")
    
    def print_summary(self):
        """Print crawling summary"""
        if not self.crawled_data:
            print("No data crawled.")
            return
        
        print(f"\nCRAWL SUMMARY")
        print("=" * 50)
        print(f"Total pages crawled: {len(self.crawled_data)}")
        print(f"Total unique URLs visited: {len(self.visited_urls)}")
        
        print(f"\nPages found:")
        for i, data in enumerate(self.crawled_data, 1):
            print(f"{i:2d}. {data['title'][:50]}...")
            print(f"     {data['url']}")
 
def main():
    # Example usage
    start_url = input("Enter the starting URL to crawl: ").strip()
    if not start_url:
        start_url = "https://example.com"
    
    try:
        max_pages = int(input("Enter maximum pages to crawl (default 5): ") or "5")
    except ValueError:
        max_pages = 5
    
    print(f"\nStarting web crawler...")
    crawler = WebCrawler(start_url, max_pages=max_pages, delay=1)
    
    try:
        crawled_data = crawler.crawl()
        crawler.print_summary()
        
        save_choice = input("\nSave results to CSV? (y/n): ").lower()
        if save_choice == 'y':
            crawler.save_to_csv()
    
    except KeyboardInterrupt:
        print("\nCrawling interrupted by user.")
    except Exception as e:
        print(f"Error during crawling: {e}")
 
if __name__ == "__main__":
    main()

Run it

run
python basicwebcrawler.py
run
python basicwebcrawler.py

The crawler prompts for a start URL, max page count, and whether to save a CSV. Sample run:

text
Start URL: https://example.com
Max pages: 5
Save CSV? (y/n): y
[1/5] Fetched https://example.com (200) — 3 new links
[2/5] Fetched https://example.com/about (200) — 1 new link
...
Saved 5 rows to crawl_results.csv
text
Start URL: https://example.com
Max pages: 5
Save CSV? (y/n): y
[1/5] Fetched https://example.com (200) — 3 new links
[2/5] Fetched https://example.com/about (200) — 1 new link
...
Saved 5 rows to crawl_results.csv

Step-by-Step Explanation

1. Imports and class skeleton

basicwebcrawler.py
from collections import deque
from urllib.parse import urljoin, urlparse
import csv
import time
import requests
from bs4 import BeautifulSoup
 
class WebCrawler:
    def __init__(self, start_url, max_pages=10, delay=1):
        self.start_url = start_url
        self.max_pages = max_pages
        self.delay = delay
        self.visited_urls = set()
        self.to_visit = deque([start_url])
        self.crawled_data = []
basicwebcrawler.py
from collections import deque
from urllib.parse import urljoin, urlparse
import csv
import time
import requests
from bs4 import BeautifulSoup
 
class WebCrawler:
    def __init__(self, start_url, max_pages=10, delay=1):
        self.start_url = start_url
        self.max_pages = max_pages
        self.delay = delay
        self.visited_urls = set()
        self.to_visit = deque([start_url])
        self.crawled_data = []
  • dequedeque (double-ended queue) gives O(1) popleft()popleft() and append()append(), which is exactly what a FIFO frontier needs.
  • visited_urlsvisited_urls is a set so duplicate detection is O(1).
  • crawled_datacrawled_data accumulates one dict per page for CSV export.

2. URL validation

basicwebcrawler.py
def is_valid_url(self, url):
    parsed = urlparse(url)
    return bool(parsed.netloc) and bool(parsed.scheme)
basicwebcrawler.py
def is_valid_url(self, url):
    parsed = urlparse(url)
    return bool(parsed.netloc) and bool(parsed.scheme)

A valid URL has both a scheme (httphttp, httpshttps) and a netloc (example.comexample.com). Without either, it cannot be fetched.

basicwebcrawler.py
def extract_links(self, html, base_url):
    soup = BeautifulSoup(html, "html.parser")
    links = []
    for a in soup.find_all("a", href=True):
        absolute = urljoin(base_url, a["href"])
        if self.is_valid_url(absolute):
            links.append(absolute)
    return links
basicwebcrawler.py
def extract_links(self, html, base_url):
    soup = BeautifulSoup(html, "html.parser")
    links = []
    for a in soup.find_all("a", href=True):
        absolute = urljoin(base_url, a["href"])
        if self.is_valid_url(absolute):
            links.append(absolute)
    return links

urljoin(base, href)urljoin(base, href) turns relative links like /about/about into absolute URLs like https://example.com/abouthttps://example.com/about. Always do this — relative paths are useless to requests.getrequests.get.

4. Extract page data

basicwebcrawler.py
def extract_page_data(self, html, url):
    soup = BeautifulSoup(html, "html.parser")
    title = soup.title.get_text(strip=True) if soup.title else ""
    meta = soup.find("meta", attrs={"name": "description"})
    description = meta["content"] if meta and meta.get("content") else ""
    headings = [h.get_text(strip=True) for h in soup.find_all(["h1", "h2", "h3"])]
    return {
        "url": url,
        "title": title,
        "description": description,
        "headings": " | ".join(headings)[:500],
    }
basicwebcrawler.py
def extract_page_data(self, html, url):
    soup = BeautifulSoup(html, "html.parser")
    title = soup.title.get_text(strip=True) if soup.title else ""
    meta = soup.find("meta", attrs={"name": "description"})
    description = meta["content"] if meta and meta.get("content") else ""
    headings = [h.get_text(strip=True) for h in soup.find_all(["h1", "h2", "h3"])]
    return {
        "url": url,
        "title": title,
        "description": description,
        "headings": " | ".join(headings)[:500],
    }

This grabs the bits most analysts care about: title, meta description, headings.

5. The main loop

basicwebcrawler.py
def crawl(self):
    while self.to_visit and len(self.visited_urls) < self.max_pages:
        url = self.to_visit.popleft()
        if url in self.visited_urls:
            continue
        try:
            response = requests.get(url, timeout=10, headers={
                "User-Agent": "BasicCrawler/1.0 (+contact@example.com)"
            })
            response.raise_for_status()
        except requests.RequestException as e:
            print(f"Failed {url}: {e}")
            continue
 
        self.visited_urls.add(url)
        self.crawled_data.append(self.extract_page_data(response.text, url))
 
        for link in self.extract_links(response.text, url):
            if urlparse(link).netloc == urlparse(self.start_url).netloc:
                if link not in self.visited_urls:
                    self.to_visit.append(link)
 
        time.sleep(self.delay)
basicwebcrawler.py
def crawl(self):
    while self.to_visit and len(self.visited_urls) < self.max_pages:
        url = self.to_visit.popleft()
        if url in self.visited_urls:
            continue
        try:
            response = requests.get(url, timeout=10, headers={
                "User-Agent": "BasicCrawler/1.0 (+contact@example.com)"
            })
            response.raise_for_status()
        except requests.RequestException as e:
            print(f"Failed {url}: {e}")
            continue
 
        self.visited_urls.add(url)
        self.crawled_data.append(self.extract_page_data(response.text, url))
 
        for link in self.extract_links(response.text, url):
            if urlparse(link).netloc == urlparse(self.start_url).netloc:
                if link not in self.visited_urls:
                    self.to_visit.append(link)
 
        time.sleep(self.delay)

Key points:

  • Timeout keeps a hanging server from freezing the crawl.
  • raise_for_status()raise_for_status() turns 4xx/5xx responses into exceptions.
  • Domain check keeps us from accidentally crawling Wikipedia from example.comexample.com.
  • time.sleep(self.delay)time.sleep(self.delay) is the polite delay between requests.

6. CSV export

basicwebcrawler.py
def save_to_csv(self, filename="crawl_results.csv"):
    if not self.crawled_data:
        return
    fieldnames = self.crawled_data[0].keys()
    with open(filename, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=fieldnames)
        writer.writeheader()
        writer.writerows(self.crawled_data)
basicwebcrawler.py
def save_to_csv(self, filename="crawl_results.csv"):
    if not self.crawled_data:
        return
    fieldnames = self.crawled_data[0].keys()
    with open(filename, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=fieldnames)
        writer.writeheader()
        writer.writerows(self.crawled_data)

csv.DictWritercsv.DictWriter matches dictionary keys to columns automatically.

Respecting robots.txtrobots.txt

Politeness is not optional. Use Python’s built-in urllib.robotparserurllib.robotparser:

robots.py
from urllib.robotparser import RobotFileParser
 
rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
 
if rp.can_fetch("BasicCrawler/1.0", url):
    response = requests.get(url, ...)
else:
    print(f"Skipping {url} — disallowed by robots.txt")
robots.py
from urllib.robotparser import RobotFileParser
 
rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
 
if rp.can_fetch("BasicCrawler/1.0", url):
    response = requests.get(url, ...)
else:
    print(f"Skipping {url} — disallowed by robots.txt")

Most public sites set their crawler rules in /robots.txt/robots.txt. Ignoring those rules is rude and can get your IP banned.

Common Mistakes

ProblemCauseFix
Crawler runs foreverNo max-pages limitAlways cap len(visited_urls) < max_pageslen(visited_urls) < max_pages
403 Forbidden everywhereDefault requestsrequests User-AgentSet a descriptive User-AgentUser-Agent header
Same page visited many timesTrailing slash or query string differencesNormalize URLs before adding to the visited set
Memory grows unboundedStoring every page’s HTMLStore only the extracted fields
RecursionErrorRecursionErrorRecursive crawl instead of iterativeUse the dequedeque-based loop above
Server returns 429Hitting too fastIncrease the delay or back off on 429

Variations to Try

1. Depth-first vs. breadth-first

Swap popleft()popleft() for pop()pop() and the queue becomes a stack — depth-first crawling. Useful for very narrow, deep sites.

2. Concurrent crawling

Use concurrent.futures.ThreadPoolExecutorconcurrent.futures.ThreadPoolExecutor to fetch several URLs in parallel — but rate-limit per domain. Anything else is impolite.

3. Sitemap-aware crawling

Fetch /sitemap.xml/sitemap.xml first to seed the frontier with URLs the site wants indexed.

4. Async crawler

Rewrite with aiohttpaiohttp + asyncioasyncio for higher throughput at the same politeness budget.

5. Download files

Extend the crawler to grab .pdf.pdf, .jpg.jpg, etc. Save under a folder named after the domain.

6. Headless browser

Some sites render content with JavaScript and serve empty HTML to requestsrequests. Use Playwright or Selenium to render the page first.

7. Persistent state

Save visited_urlsvisited_urls and to_visitto_visit to a JSON file every minute so you can resume after a crash.

8. Incremental indexing

After each fetch, push the page text into a SQLite full-text index. You now have a personal Google.

Crawling Ethics & Legality

Crawling is powerful and risky. Before you point your crawler at a real site:

  • Read its Terms of Service. Many sites explicitly forbid automated access.
  • Honor robots.txtrobots.txt.
  • Avoid crawling content behind login forms unless you have permission.
  • Set a polite delay — at least 1 second between requests by default.
  • Identify your crawler with a real User-AgentUser-Agent and contact URL.
  • Stop immediately if the site asks you to.

Hammering a small site can cost the owner real money in bandwidth bills. Treat every fetch like a tiny favor someone is doing you.

Real-World Applications

  • Search engines — Google’s crawler is the most famous example.
  • Price-tracking bots — comparing items across e-commerce sites (with permission).
  • SEO audits — building a sitemap, finding broken internal links.
  • News aggregators — collecting recent headlines for analysis.
  • Academic research — gathering corpora for NLP training (where licenses allow).

Educational Value

This project teaches:

  • HTTP — requests, status codes, headers.
  • HTML parsing with BeautifulSoup.
  • URL manipulation with urllib.parseurllib.parse.
  • Data structures — queues for the frontier, sets for the visited list.
  • OOP — wrapping state and behavior in a class.
  • Ethics — politeness is engineering, not a nice-to-have.

Next Steps

  • Combine with RSS Feed Reader to crawl articles automatically.
  • Add content classification with a small ML model (scikit-learnscikit-learn).
  • Build a search interface with Flask (see Basic Web Server).
  • Deploy as a scheduled job — re-crawl every day, diff the results, alert on changes.
  • Move to a real crawler framework like Scrapy when your needs outgrow a single script.

Conclusion

You wrote a real, working web crawler — not a toy. It fetches pages politely, parses them, follows links, stays on-domain, and exports structured results. The patterns scale: a thousand-line Scrapy spider is still doing the same three things in a loop. Full source on GitHub. Explore more web projects on Python Central Hub.

If this helped you, consider buying me a coffee ☕

Buy me a coffee

Was this page helpful?

Let us know how we did