Introduction to Web Scraping Ethics & robots.txt

  flowchart TD
  A["you want data from a site"] --> B{"is there an API?"}
  B -->|yes| C["use it -- stable, documented, allowed"]
  B -->|no| D["read /robots.txt"]
  D --> E{"is the path disallowed?"}
  E -->|yes| F["do not scrape it"]
  E -->|no| G["check the terms of service"]
  G --> H["identify yourself in the User-Agent"]
  H --> I["rate limit -- one request at a time, with a delay"]
  I --> J["cache responses so a re-run does not re-fetch"]
  J --> K["stop on 429 or 403 and back off"]

Scraping principles

• Prefer official APIs when available
• Respect Terms of Service
• Rate-limit requests
• Identify your scraper (User-Agent)
• Don’t scrape private data

robots.txt basics

robots.txt is a convention that tells crawlers which paths are allowed/disallowed.

It’s not a security feature, but a strong signal.

Rate limiting

Use delays and backoff:

import time
import random
 
 
def polite_sleep(base=1.0):
    time.sleep(base + random.random())

Avoid getting blocked

• keep concurrency low
• cache responses
• handle 429/503
• rotate proxies only if permitted and ethical

Why this comes before the code

Scraping puts load on someone else's server and often touches content someone else owns. Being able to fetch a page does not make it acceptable to fetch it a million times, or to reuse what you get. The checks in the diagram above are quick, and most problems are avoided by doing them first.

Checking robots.txt in Python

Every site can publish https://site/robots.txt. It lists which paths crawlers may or may not request, and sometimes a Crawl-delay. It is a convention, not a lock: nothing forces a scraper to obey it, but you should. Python's standard library reads it for you.

from urllib.robotparser import RobotFileParser
 
rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()                                   # downloads and parses the file
 
agent = "Python Central HubBot"
url = "https://example.com/private/data"
if rp.can_fetch(agent, url):
    print("allowed")
else:
    print("disallowed, skip it")
 
print("crawl delay:", rp.crawl_delay(agent))

can_fetch answers for one user-agent and one URL. crawl_delay returns the requested seconds between requests, or None. Call read() once and reuse the parser, rather than downloading robots.txt before every page.

You can also give the parser the file's lines directly with rp.parse(lines), which is how the exercise below tests it without a network.

Being a good client

• Identify yourself. Put a real name and a contact address in the User-Agent, such as MyResearchBot/1.0 (you@example.com), so a site owner can reach you rather than block you.
• Slow down. One request at a time with a delay, and longer if the site sends 429 or 503. Read the Retry-After header when it is present.
• Cache. Save each page you fetch, so re-running the script while debugging does not fetch it again.
• Take only what you need. Do not collect personal data. Laws such as GDPR apply to data about people even if it is public.
• Check the terms. Some sites forbid automated access outright, whatever robots.txt says. Copyright still applies to content you download.
• Prefer an API. It is documented, stable, and allowed.

Common mistakes

• Treating "not blocked" as "allowed".
• Ignoring 429 and 403. Keep hammering and you will be banned, or worse.
• Hiding your identity with a fake browser User-Agent to bypass rules.
• Scraping behind a login that you do not own.
• Concurrency too high. Ten parallel workers is ten times the load.

🧪 Try It Yourself

Exercise 1 – Test robots.txt Rules

Press Run to see what your code prints.

Check your understanding