Working with JSON APIs

What is JSON?

JSON is a text format for structured data.

objects → Python dict
arrays → Python list

Parse JSON

parse_json.py

import requests
 
r = requests.get("https://api.github.com", timeout=10)
obj = r.json()
 
print(type(obj))
print(obj.get("current_user_url"))

Validate fields safely

APIs may change or fields may be missing.

safe_access.py

data = {"user": {"name": "Ravi"}}
 
name = data.get("user", {}).get("name")
email = data.get("user", {}).get("email")  # might be None
 
print(name, email)

Convert JSON list to pandas

json_to_df.py

import requests
import pandas as pd
 
url = "https://api.github.com/search/repositories"
params = {"q": "data analytics", "per_page": 10}
 
r = requests.get(url, params=params, timeout=10)
r.raise_for_status()
 
items = r.json()["items"]
rows = [
    {
        "name": it["full_name"],
        "stars": it["stargazers_count"],
        "language": it.get("language"),
    }
    for it in items
]
 
df = pd.DataFrame(rows)
print(df.head())

Common errors

KeyError: field missing → use .get()
JSONDecodeError: response not JSON → check status code/content

The round trip is not lossless

JSON has six types. Python has many more. Everything you send is mapped onto the nearest JSON type, and what comes back is that JSON type — not what you started with:

diagram Diagram mermaid

  flowchart LR
  subgraph PY["Python"]
    T["tuple (1, 2)"]
    IK["dict key 1 (int)"]
    S["set {1, 2}"]
    D["datetime"]
  end
  subgraph JS["JSON"]
    A["array [1, 2]"]
    SK["string key '1'"]
    X["no equivalent"]
  end
  subgraph BACK["after loads()"]
    L["list [1, 2]"]
    SK2["string key '1'"]
  end
  T --> A --> L
  IK --> SK --> SK2
  S --> X
  D --> X

round_trip.py

import json
 
s = json.dumps({"tup": (1, 2), "intkey": {1: "a"}})
print(s)                     # {"tup": [1, 2], "intkey": {"1": "a"}}
 
back = json.loads(s)
print(type(back["tup"]).__name__)          # list   <- was a tuple
print(list(back["intkey"])[0])             # '1'    <- was the integer 1
 
json.dumps({"s": {1, 2}})    # TypeError: Object of type set is not JSON serializable

Two of these are silent and one is loud:

A tuple becomes a list. No warning.
An integer dict key becomes a string. No warning — and back["intkey"][1] now raises KeyError.
A set raises TypeError. Convert it with sorted(...) or list(...) first.
datetime also raises. Send ISO strings with .isoformat().

Numbers deserve a second look

numbers.py

json.loads("1")        # 1     (int)
json.loads("1.0")      # 1.0   (float)   <- the decimal point decides the type
json.loads("1e400")    # inf             <- silently, no error
 
json.dumps(10**25)     # '10000000000000000000000000'   Python round-trips it exactly
# but 2**53 = 9007199254740992 is the largest integer a JavaScript client
# can represent exactly, so a large id may be corrupted by the *consumer*

1e400 overflowing to inf without complaint is the dangerous one — you get a float that poisons every later calculation. And note the asymmetry: Python handles huge integers fine, so an id that survives your tests can still be mangled by a browser reading the same payload. APIs send large ids as strings for exactly this reason.

NaN is a related trap: json.dumps(float("nan")) produces NaN, which Python reads back happily and which is not valid JSON — strict parsers in other languages reject it.

See it move

Click a Python value to send it through dumps and back. Watch which ones come home unchanged, which quietly change type, and which refuse to go at all.

sketch What survives a JSON round trip p5.js

Tuples return as lists and integer keys as strings, both silently. Sets and datetimes raise. Only str, int, float, bool, None, list and dict survive unchanged.

Reading a response defensively

Real payloads have optional fields, and [] raises on the first one that is missing:

safe_access.py

r = {"user": {"name": "ada"}}
 
r["user"]["email"]                     # KeyError: 'email'
r["user"].get("email")                 # None
r.get("account", {}).get("email")      # None   <- safe even though 'account' is absent

The .get("key", {}) chain is the idiom worth memorising: supplying an empty dict as the default keeps the chain going instead of raising halfway through.

For anything deeper than two levels, validate the shape once at the boundary rather than sprinkling .get everywhere — a dataclass, or a library like pydantic, turns “this field was missing” into one clear error at the point of parsing.

Writing JSON you will be glad to read

dumps_options.py

obj = {"b": 2, "a": [1, 2, 3]}
 
json.dumps(obj)                              # {"b": 2, "a": [1, 2, 3]}      24 bytes
json.dumps(obj, separators=(",", ":"))       # {"b":2,"a":[1,2,3]}           19 bytes
json.dumps(obj, sort_keys=True)              # {"a": [1, 2, 3], "b": 2}
json.dumps({"city": "Kraków"})               # {"city": "Krak\u00f3w"}       23 bytes
json.dumps({"city": "Kraków"}, ensure_ascii=False)   # {"city": "Kraków"}    19 bytes

separators=(",", ":") for anything going over a network — 19 bytes against 24 on this tiny object, and the gap widens with size.
sort_keys=True for anything written to a file and compared later; without it a diff is noise.
ensure_ascii=False to keep text readable. Both forms decode to the same string; the escaped version is only larger and harder to read.

Check yourself

pch.quizTag pch.quizDefaultTitle

You json.dumps a dict with the integer key 1, then json.loads it back. What key does the result have?

pch.quizShowAnswer

B — the string '1', because JSON object keys are always strings — JSON keys can only be strings, so dumps converts silently. Code that then looks up data[1] raises KeyError on data that prints as if it were fine.
What does json.loads('1e400') return?

pch.quizShowAnswer

C — inf, silently — It overflows to float infinity with no error, and inf then contaminates every later calculation. Validate numeric ranges when the source is not yours.
Which of these raises TypeError when passed to json.dumps?

pch.quizShowAnswer

C — a set — A set has no JSON equivalent and raises 'Object of type set is not JSON serializable'. A tuple does not raise — it silently becomes an array, which is arguably worse.
Given r = {'user': {'name': 'ada'}}, which expression safely yields None instead of raising?

pch.quizShowAnswer

C — r.get('account', {}).get('email') — Supplying {} as the default keeps the chain alive when 'account' is missing. The other three all hit a missing key or call .get on None.