HTTP Requests with the requests Library

  flowchart TD
  A["requests.get(url)"] --> B{"timeout= given?"}
  B -->|no| C["can block forever -- there is no default"]
  B -->|yes| D["response, or an exception in bounded time"]
  D --> E{"check the status"}
  E -->|"2xx"| F["parse the body"]
  E -->|"4xx / 5xx"| G["r.ok is False, but nothing was raised"]
  G --> H["call raise_for_status() or check explicitly"]
  H --> I{"retry?"}
  I -->|yes| J["back off -- doubling delay, with a cap"]
  I -->|no| K["log the status and the URL"]
  L["many calls to one host"] --> M["use a Session: connection reuse + shared headers"]

Always use timeouts

import requests
 
r = requests.get("https://httpbin.org/get", timeout=10)
r.raise_for_status()
print(r.status_code)
print(r.json())

Headers and User-Agent

import requests
 
headers = {"User-Agent": "Python Central HubBot/1.0"}
r = requests.get("https://httpbin.org/headers", headers=headers, timeout=10)
print(r.json())

Sessions (cookies + connection pooling)

import requests
 
with requests.Session() as s:
    s.headers.update({"User-Agent": "MyBot/1.0"})
    r = s.get("https://httpbin.org/cookies/set?hello=world", timeout=10)
    r.raise_for_status()
 
    r2 = s.get("https://httpbin.org/cookies", timeout=10)
    print(r2.json())

The basics of a request

An HTTP request has a method (GET to read, POST to send data), a URL, optional headers, and optionally a body. The response has a status code, headers, and a body. requests maps these to short calls.

• requests.get(url, params={"q": "python"}) builds the query string ?q=python and encodes it for you.
• requests.post(url, json={"name": "Ana"}) sends a JSON body and sets the Content-Type header. Use data={...} instead to send a form.
• r.status_code is the number, r.ok is True for codes below 400, r.text is the body as text, and r.json() parses a JSON body.

The flowchart above states the rule that matters most in a script: nobody is watching, so the code must handle failure. requests has no default timeout, so a stalled server can block your script forever. Always pass timeout=. And a 404 or 500 does not raise an exception by itself. Call r.raise_for_status() to make it do so.

A resilient helper

This function combines a timeout, status checking, and automatic retries with a growing delay, using the retry support that requests already includes through urllib3.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
 
 
def make_session() -> requests.Session:
    retry = Retry(
        total=3,
        backoff_factor=1.0,                       # waits 1s, 2s, 4s between tries
        status_forcelist=[429, 500, 502, 503, 504],
        allowed_methods=["GET"],                  # do not blindly retry POST
    )
    session = requests.Session()
    session.mount("https://", HTTPAdapter(max_retries=retry))
    session.mount("http://", HTTPAdapter(max_retries=retry))
    session.headers.update({"User-Agent": "Python Central HubBot/1.0"})
    return session
 
 
with make_session() as s:
    try:
        r = s.get("https://httpbin.org/get", params={"q": "python"}, timeout=10)
        r.raise_for_status()
        print(r.json()["args"])
    except requests.Timeout:
        print("timed out")
    except requests.HTTPError as err:
        print("bad status:", err.response.status_code)
    except requests.RequestException as err:
        print("request failed:", err)

requests.RequestException is the parent of all the others, so catch the specific errors first and it last. Retrying only GET matters: repeating a POST could create a duplicate order or send an email twice.

Sessions

A Session keeps cookies and reuses the TCP connection between calls. That makes repeated requests to one host faster, and it lets you log in once and stay logged in. It also holds shared headers, so you set the User-Agent once.

Common mistakes

• No timeout. Pass one to every call, for example timeout=10 or (3, 10) for connect and read.
• Assuming success. Check the status or call raise_for_status().
• Calling r.json() on an HTML error page. It raises JSONDecodeError. Check the status and Content-Type first.
• Putting secrets in URLs. Tokens in query strings end up in logs. Send them in headers.
• Retrying without a delay, which turns one failure into a burst of requests.

Adapting it

Use the same session for an API that needs a token (session.headers["Authorization"] = f"Bearer {token}"), and read the token from an environment variable, never from the source file.

🧪 Try It Yourself

Exercise 1 – List Files with os.listdir

Press Run to see what your code prints.

Exercise 2 – Join Paths with os.path.join

Press Run to see what your code prints.

Exercise 3 – Write and Read a File

Press Run to see what your code prints.