Working with JSON APIs

What is JSON?

JSON is a text format for structured data.

• objects → Python dict
• arrays → Python list

Parse JSON

import requests
 
r = requests.get("https://api.github.com", timeout=10)
obj = r.json()
 
print(type(obj))
print(obj.get("current_user_url"))

Validate fields safely

APIs may change or fields may be missing.

data = {"user": {"name": "Ravi"}}
 
name = data.get("user", {}).get("name")
email = data.get("user", {}).get("email")  # might be None
 
print(name, email)

Convert JSON list to pandas

import requests
import pandas as pd
 
url = "https://api.github.com/search/repositories"
params = {"q": "data analytics", "per_page": 10}
 
r = requests.get(url, params=params, timeout=10)
r.raise_for_status()
 
items = r.json()["items"]
rows = [
    {
        "name": it["full_name"],
        "stars": it["stargazers_count"],
        "language": it.get("language"),
    }
    for it in items
]
 
df = pd.DataFrame(rows)
print(df.head())

Common errors

• KeyError: field missing → use .get()
• JSONDecodeError: response not JSON → check status code/content

The round trip is not lossless

JSON has six types. Python has many more. Everything you send is mapped onto the nearest JSON type, and what comes back is that JSON type — not what you started with:

  flowchart LR
  subgraph PY["Python"]
    T["tuple (1, 2)"]
    IK["dict key 1 (int)"]
    S["set {1, 2}"]
    D["datetime"]
  end
  subgraph JS["JSON"]
    A["array [1, 2]"]
    SK["string key '1'"]
    X["no equivalent"]
  end
  subgraph BACK["after loads()"]
    L["list [1, 2]"]
    SK2["string key '1'"]
  end
  T --> A --> L
  IK --> SK --> SK2
  S --> X
  D --> X

import json
 
s = json.dumps({"tup": (1, 2), "intkey": {1: "a"}})
print(s)                     # {"tup": [1, 2], "intkey": {"1": "a"}}
 
back = json.loads(s)
print(type(back["tup"]).__name__)          # list   <- was a tuple
print(list(back["intkey"])[0])             # '1'    <- was the integer 1
 
json.dumps({"s": {1, 2}})    # TypeError: Object of type set is not JSON serializable

Two of these are silent and one is loud:

• A tuple becomes a list. No warning.
• An integer dict key becomes a string. No warning — and back["intkey"][1] now raises KeyError.
• A set raises TypeError. Convert it with sorted(...) or list(...) first.
• datetime also raises. Send ISO strings with .isoformat().

Numbers deserve a second look

json.loads("1")        # 1     (int)
json.loads("1.0")      # 1.0   (float)   <- the decimal point decides the type
json.loads("1e400")    # inf             <- silently, no error
 
json.dumps(10**25)     # '10000000000000000000000000'   Python round-trips it exactly
# but 2**53 = 9007199254740992 is the largest integer a JavaScript client
# can represent exactly, so a large id may be corrupted by the *consumer*

1e400 overflowing to inf without complaint is the dangerous one — you get a float that poisons every later calculation. And note the asymmetry: Python handles huge integers fine, so an id that survives your tests can still be mangled by a browser reading the same payload. APIs send large ids as strings for exactly this reason.

NaN is a related trap: json.dumps(float("nan")) produces NaN, which Python reads back happily and which is not valid JSON — strict parsers in other languages reject it.

See it move

Click a Python value to send it through dumps and back. Watch which ones come home unchanged, which quietly change type, and which refuse to go at all.

Reading a response defensively

Real payloads have optional fields, and [] raises on the first one that is missing:

r = {"user": {"name": "ada"}}
 
r["user"]["email"]                     # KeyError: 'email'
r["user"].get("email")                 # None
r.get("account", {}).get("email")      # None   <- safe even though 'account' is absent

The .get("key", {}) chain is the idiom worth memorising: supplying an empty dict as the default keeps the chain going instead of raising halfway through.

For anything deeper than two levels, validate the shape once at the boundary rather than sprinkling .get everywhere — a dataclass, or a library like pydantic, turns "this field was missing" into one clear error at the point of parsing.

Writing JSON you will be glad to read

obj = {"b": 2, "a": [1, 2, 3]}
 
json.dumps(obj)                              # {"b": 2, "a": [1, 2, 3]}      24 bytes
json.dumps(obj, separators=(",", ":"))       # {"b":2,"a":[1,2,3]}           19 bytes
json.dumps(obj, sort_keys=True)              # {"a": [1, 2, 3], "b": 2}
json.dumps({"city": "Kraków"})               # {"city": "Krak\u00f3w"}       23 bytes
json.dumps({"city": "Kraków"}, ensure_ascii=False)   # {"city": "Kraków"}    19 bytes

• separators=(",", ":") for anything going over a network — 19 bytes against 24 on this tiny object, and the gap widens with size.
• sort_keys=True for anything written to a file and compared later; without it a diff is noise.
• ensure_ascii=False to keep text readable. Both forms decode to the same string; the escaped version is only larger and harder to read.

Check yourself

🧪 Try It Yourself

Exercise 1 – Parse JSON Response

Press Run to see what your code prints.

Exercise 2 – Serialize Python to JSON

Press Run to see what your code prints.

Exercise 3 – Parse JSON String

Press Run to see what your code prints.