Introduction to Web Scraping Ethics & robots.txt

diagram the checks to make before a scraper sends its first request mermaid

  flowchart TD
  A["you want data from a site"] --> B{"is there an API?"}
  B -->|yes| C["use it -- stable, documented, allowed"]
  B -->|no| D["read /robots.txt"]
  D --> E{"is the path disallowed?"}
  E -->|yes| F["do not scrape it"]
  E -->|no| G["check the terms of service"]
  G --> H["identify yourself in the User-Agent"]
  H --> I["rate limit -- one request at a time, with a delay"]
  I --> J["cache responses so a re-run does not re-fetch"]
  J --> K["stop on 429 or 403 and back off"]

Most of what separates acceptable scraping from the other kind is decided before any code runs. robots.txt states what the site asks automated clients not to touch, the terms may say more, and an API -- if one exists -- is both kinder and more reliable than parsing HTML that was never meant as an interface.

Scraping principles

• Prefer official APIs when available
• Respect Terms of Service
• Rate-limit requests
• Identify your scraper (User-Agent)
• Don’t scrape private data

robots.txt basics

robots.txt is a convention that tells crawlers which paths are allowed/disallowed.

It’s not a security feature, but a strong signal.

Rate limiting

Use delays and backoff:

polite_delay.py

import time
import random
 
 
def polite_sleep(base=1.0):
    time.sleep(base + random.random())

Avoid getting blocked

• keep concurrency low
• cache responses
• handle 429/503
• rotate proxies only if permitted and ethical