Binary Formats and Web APIs (Parquet, pickle, requests)

Beyond CSV: faster and richer formats

CSV is human-readable but slow to parse and loses type information (everything comes back as text until pandas guesses otherwise). Once a project outgrows quick CSV exports, three tools solve different problems:

• Binary formats (pickle, Parquet, HDF5) — faster reads/writes, types preserved.
• Chunked reading (chunksize) — process files bigger than memory, piece by piece.
• Web APIs (requests + JSON) — pull data directly from a live service instead of a file at all.

Pickle: quick, Python-only serialization

to_pickle / read_pickle dump a pandas object to disk (or any file-like object) almost instantly, preserving dtypes exactly:

pickle round-trip

import pandas as pd
import io
 
df = pd.DataFrame({"a": [1, 2, 3], "b": ["x", "y", "z"]})
 
buffer = io.BytesIO()
df.to_pickle(buffer)   # in real code: df.to_pickle("data.pkl")
 
buffer.seek(0)
restored = pd.read_pickle(buffer)   # in real code: pd.read_pickle("data.pkl")
print(restored)

Pickle is a short-term format

McKinney warns that pickle is “recommended only as a short-term storage format” — a file pickled with today’s pandas version isn’t guaranteed to load correctly with a much later version. Use it for caching intermediate results within one project, not for long-term archival or sharing between different environments.

Parquet and Feather: columnar, cross-language, fast

Apache Parquet (via to_parquet / read_parquet) stores data column-by-column with compression, keeping full type information — and unlike pickle, it’s readable from other languages (R, Spark, Java) too:

Parquet (requires the pyarrow package)

# pip install pyarrow  (or: conda install pyarrow)
df.to_parquet("data.parquet")
loaded = pd.read_parquet("data.parquet")
print(loaded.dtypes)

Feather (to_feather / read_feather) is a similar columnar format optimized for very fast reads/writes, most useful for short-lived caches between processing steps.

HDF5: many datasets in one file

HDF5 (via pandas.HDFStore) is built for large arrays that don’t fit in memory — you can efficiently read a slice of a huge file without loading all of it:

HDFStore (requires the tables package)

# pip install tables
store = pd.HDFStore("mydata.h5")
store["prices"] = df          # write, dictionary-style
print(store["prices"])        # read back, dictionary-style
store.close()

HDF5 is best for “write-once, read-many” workloads — it isn’t a database, and concurrent writers can corrupt the file.

Reading huge files in chunks

For files too large to comfortably fit in memory, chunksize turns read_csv into an iterator — you process the file piece by piece instead of all at once:

Aggregating a file in chunks

import io
 
csv_text = "key,amount\nA,10\nB,20\nA,30\nC,40\nB,50\nA,60\n"
chunker = pd.read_csv(io.StringIO(csv_text), chunksize=2)  # 2 rows per chunk
 
totals = pd.Series([], dtype="int64")
for piece in chunker:
    totals = totals.add(piece.groupby("key")["amount"].sum(), fill_value=0)
 
print(totals.sort_values(ascending=False))

Each piece is a normal, small DataFrame — you aggregate it and merge the running total with .add(..., fill_value=0), so no key is ever lost between chunks.

Pulling JSON from a web API

Many services expose data as a JSON HTTP response. The requests library fetches it; resp.json() parses it into plain Python dicts/lists, which pd.DataFrame can consume directly:

requests + DataFrame (illustrative — needs network access)

import requests
 
url = "https://api.github.com/repos/pandas-dev/pandas/issues"
resp = requests.get(url)
resp.raise_for_status()  # raise an error early if the request failed
 
data = resp.json()  # list of dicts, one per issue
issues = pd.DataFrame(data, columns=["number", "title", "state"])
print(issues.head())

The exercises below simulate the parsed JSON as a plain Python list of dicts (exactly what resp.json() would hand you), so you can practice the DataFrame-building step without needing live network access.

Common pitfalls

• Parquet and HDF5 both need an extra package installed (pyarrow/fastparquet, tables) — a missing package produces a clear ImportError, not a silent failure.
• Always call resp.raise_for_status() after requests.get() — otherwise a failed request (like a 404) can silently hand you an error page instead of real data.
• When processing chunks, remember each chunk is aggregated independently — you must combine the partial results yourself (as with .add(..., fill_value=0) above).

Visualize it

diagram Choosing an I/O path mermaid

  flowchart TD
  A["I need to load or save data"] --> B{"Where does it live?"}
  B -->|"Local, short-term cache"| C["pickle
to_pickle / read_pickle"]
  B -->|"Local, long-term / cross-language"| D["Parquet
to_parquet / read_parquet"]
  B -->|"Huge array, partial reads"| E["HDF5
HDFStore"]
  B -->|"Huge CSV, won't fit in memory"| F["chunksize
iterate and aggregate"]
  B -->|"Lives on a remote web service"| G["requests + read_json
resp.json() -> DataFrame"]

Different storage needs point to different tools — this is the decision flow behind picking pickle, Parquet, HDF5, chunked CSV reads, or a web API.

From the book

McKinney recommends always calling resp.raise_for_status() right after requests.get() — it’s a cheap, one-line habit that turns a silent bad response (like a 404 or rate-limit error) into an immediate, obvious exception instead of a confusing empty or malformed DataFrame further down your script.

🧪 Try It Yourself

Exercise 1 – Round-Trip a DataFrame Through Pickle

pch.exerciseTag pch.exerciseTitle

Exercise 2 – Aggregate a File in Chunks

pch.exerciseTag pch.exerciseTitle

Exercise 3 – Turn a Parsed JSON Response Into a DataFrame

pch.exerciseTag pch.exerciseTitle

Next

With data loading covered end to end, move on to Correlation and Covariance to start measuring relationships between the columns you’ve loaded.