HTTP Requests with the requests Library

diagram the four things every scripted HTTP call needs mermaid

  flowchart TD
  A["requests.get(url)"] --> B{"timeout= given?"}
  B -->|no| C["can block forever -- there is no default"]
  B -->|yes| D["response, or an exception in bounded time"]
  D --> E{"check the status"}
  E -->|"2xx"| F["parse the body"]
  E -->|"4xx / 5xx"| G["r.ok is False, but nothing was raised"]
  G --> H["call raise_for_status() or check explicitly"]
  H --> I{"retry?"}
  I -->|yes| J["back off -- doubling delay, with a cap"]
  I -->|no| K["log the status and the URL"]
  L["many calls to one host"] --> M["use a Session: connection reuse + shared headers"]

A request in a script is not a request in a browser: there is nobody watching it, so it has to handle its own failure. A missing timeout can hang forever, a 4xx does not raise on its own, and a retry without a delay turns one failure into a burst. A Session is what makes repeated calls to the same host both faster and consistent.

Always use timeouts

requests_get.py

import requests
 
r = requests.get("https://httpbin.org/get", timeout=10)
r.raise_for_status()
print(r.status_code)
print(r.json())

Headers and User-Agent

requests_headers.py

import requests
 
headers = {"User-Agent": "PythonCentralHubBot/1.0"}
r = requests.get("https://httpbin.org/headers", headers=headers, timeout=10)
print(r.json())

Sessions (cookies + connection pooling)

requests_session.py

import requests
 
with requests.Session() as s:
    s.headers.update({"User-Agent": "MyBot/1.0"})
    r = s.get("https://httpbin.org/cookies/set?hello=world", timeout=10)
    r.raise_for_status()
 
    r2 = s.get("https://httpbin.org/cookies", timeout=10)
    print(r2.json())

🧪 Try It Yourself

Exercise 1 – List Files with os.listdir

pch.exerciseTag pch.exerciseTitle

Exercise 2 – Join Paths with os.path.join

pch.exerciseTag pch.exerciseTitle

Exercise 3 – Write and Read a File

pch.exerciseTag pch.exerciseTitle