HTTP Requests with requests

What is an HTTP request?

An HTTP request asks a server for something.

Common verbs:

GET: fetch data
POST: send data
PUT/PATCH: update
DELETE: remove

GET request

get_request.py

import requests
 
url = "https://api.github.com"
r = requests.get(url, timeout=10)
 
print(r.status_code)
print(r.headers.get("content-type"))
print(r.json().keys())

Query parameters

params.py

import requests
 
url = "https://api.github.com/search/repositories"
params = {"q": "python", "sort": "stars", "per_page": 5}
 
r = requests.get(url, params=params, timeout=10)
r.raise_for_status()
 
data = r.json()
print("count:", data["total_count"])
for item in data["items"]:
    print(item["full_name"], item["stargazers_count"])

POST JSON

post_json.py

import requests
 
url = "https://httpbin.org/post"
payload = {"name": "ravi", "role": "learner"}
 
r = requests.post(url, json=payload, timeout=10)
r.raise_for_status()
 
print(r.json()["json"])  # echoed back

Timeouts and retries

Always set timeouts

Without a timeout, a request can hang forever.

Basic retry pattern

retry_pattern.py

import time
import requests
 
url = "https://httpbin.org/status/503"
 
for attempt in range(3):
    try:
        r = requests.get(url, timeout=5)
        r.raise_for_status()
        print("success")
        break
    except requests.RequestException as e:
        print("attempt failed:", attempt + 1, e)
        time.sleep(1)
else:
    print("all retries failed")

Good practice checklist

Use timeout=
Call raise_for_status() for non-2xx responses
Don’t log secrets (API keys)

A bad status is not an exception

This trips up almost everyone once. requests.get raises only when it could not complete the exchange. A server that answers “404” answered — the call succeeds:

diagram Diagram mermaid

  flowchart TD
  G["requests.get(url)"] --> C{"could the exchange complete?"}
  C -->|"DNS fails, refused, TLS error"| CE["raises ConnectionError"]
  C -->|"no reply within timeout"| TE["raises Timeout"]
  C -->|"a reply arrived"| OK["returns a Response
NO exception, whatever the status"]
  OK --> S{"r.status_code"}
  S -->|"2xx"| G2["r.ok is True"]
  S -->|"4xx / 5xx"| B["r.ok is False
you must check, or call raise_for_status()"]

Measured against a local endpoint that returns 404:

not_an_error.py

r = requests.get(f"{BASE}/404")
 
print(r.status_code)     # 404
print(r.ok)              # False
print(bool(r))           # False   <- `if r:` works, and is easy to forget
print(r.json())          # {'error': 'not found'}   <- the body still parses
 
r.raise_for_status()     # requests.HTTPError: 404 Client Error: Not Found for url: ...

bool(r) following r.ok is the detail worth remembering: if response: means did it succeed, not did I get a response. A truthiness test on a 404 is False even though the object is perfectly real.

`.json()` fails on anything that is not JSON

json_fail.py

r = requests.get(f"{BASE}/html")     # Content-Type: text/html
r.json()
# requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
 
r.text        # '<html><body>not json</body></html>'   <- always available

The error message points at “line 1 column 1” because the very first character was <. When an API misbehaves, that message almost always means you received an HTML error page — a login redirect, a proxy notice, or a gateway error — not malformed JSON.

Let `requests` build the URL

params.py

r = requests.get(f"{BASE}/search", params={"q": "a b&c", "page": 2})
print(r.url)
# http://127.0.0.1:61392/search?q=a+b%26c&page=2

The space became + and the & became %26, so the value cannot break out and invent a new parameter. Building query strings with f-strings is how injection bugs and mysterious truncated values happen.

headers.py

r = requests.get(url, headers={"X-Demo": "hello"})   # server saw 'hello'

See it move

A request has phases, and a timeout can be set on each. Drag the limit and watch which phase gets cut off — this is the difference between “the site is slow” and “the site is down”.

sketch Where a timeout applies p5.js

A request resolves DNS, connects, sends, waits for the server, then reads the body. requests takes (connect, read) timeouts; a single number applies to both.

Measured against an endpoint that sleeps 3 s:

call	outcome
`requests.get(url, timeout=0.5)`	`ReadTimeout` after 0.51 s
`requests.get(url, timeout=10)`	`200` after 3.01 s

Text, bytes, and encoding

body.py

r.text        # str   — decoded using r.encoding ('utf-8' here)
r.content     # bytes — exactly what arrived, 36 bytes

Use .content for anything that is not text — images, zip files, PDFs. Use .text when you want a string, and set r.encoding yourself if the server’s declaration is wrong.

Check yourself

pch.quizTag pch.quizDefaultTitle

requests.get returns a response with status 404. What happens?

pch.quizShowAnswer

B — nothing is raised; r.status_code is 404, r.ok is False, and r.json() still parses the body — The exchange completed, so requests returns normally. Only transport failures raise. Use raise_for_status() or check r.ok to turn a bad status into an error.
r.json() raises JSONDecodeError: Expecting value: line 1 column 1 (char 0). What is the most likely cause?

pch.quizShowAnswer

B — the response body is not JSON at all, typically an HTML error or login page — Failing at the very first character means the body did not start like JSON. Print r.text and r.headers['Content-Type'] — it is usually HTML from a proxy, redirect or gateway.
What does requests.get(url, params={'q': 'a b&c', 'page': 2}) produce as a URL?

pch.quizShowAnswer

B — ...?q=a+b%26c&page=2, with the space and ampersand encoded — requests percent-encodes values so a stray & cannot invent a new parameter. This is why building query strings by hand with f-strings is a bug waiting to happen.
What is the default timeout for requests.get?

pch.quizShowAnswer

C — there is none; the call can block indefinitely — requests has no default timeout, so an unresponsive server hangs the script silently. Measured: timeout=0.5 raised ReadTimeout after 0.51 s; without one there is nothing to stop the wait.

HTTP Requests with requests

What is an HTTP request?

GET request

Query parameters

POST JSON

Timeouts and retries

Always set timeouts

Basic retry pattern

Good practice checklist

A bad status is not an exception

`.json()` fails on anything that is not JSON

Let `requests` build the URL

See it move

Text, bytes, and encoding

Check yourself

🧪 Try It Yourself

Exercise 1 – GET Request

Exercise 2 – POST with JSON

Exercise 3 – Handle Timeout

pch.feedbackHeading

HTTP Requests with requests

What is an HTTP request?

GET request

Query parameters

POST JSON

Timeouts and retries

Always set timeouts

Basic retry pattern

Good practice checklist

A bad status is not an exception

.json() fails on anything that is not JSON

Let requests build the URL

See it move

Text, bytes, and encoding

Check yourself

🧪 Try It Yourself

Exercise 1 – GET Request

Exercise 2 – POST with JSON

Exercise 3 – Handle Timeout

pch.feedbackHeading

`.json()` fails on anything that is not JSON

Let `requests` build the URL