Binary Formats and Web APIs (Parquet, pickle, requests)

Beyond CSV: faster and richer formats

CSV is human-readable but slow to parse and loses type information (everything comes back as text until pandas guesses otherwise). Once a project outgrows quick CSV exports, three tools solve different problems:

• Binary formats (pickle, Parquet, HDF5) — faster reads/writes, types preserved.
• Chunked reading (chunksize) — process files bigger than memory, piece by piece.
• Web APIs (requests + JSON) — pull data directly from a live service instead of a file at all.

Pickle: quick, Python-only serialization

to_pickle / read_pickle dump a pandas object to disk (or any file-like object) almost instantly, preserving dtypes exactly:

import pandas as pd
import io
 
df = pd.DataFrame({"a": [1, 2, 3], "b": ["x", "y", "z"]})
 
buffer = io.BytesIO()
df.to_pickle(buffer)   # in real code: df.to_pickle("data.pkl")
 
buffer.seek(0)
restored = pd.read_pickle(buffer)   # in real code: pd.read_pickle("data.pkl")
print(restored)

Parquet and Feather: columnar, cross-language, fast

Apache Parquet (via to_parquet / read_parquet) stores data column-by-column with compression, keeping full type information — and unlike pickle, it's readable from other languages (R, Spark, Java) too:

# pip install pyarrow  (or: conda install pyarrow)
df.to_parquet("data.parquet")
loaded = pd.read_parquet("data.parquet")
print(loaded.dtypes)

Feather (to_feather / read_feather) is a similar columnar format optimized for very fast reads/writes, most useful for short-lived caches between processing steps.

HDF5: many datasets in one file

HDF5 (via pandas.HDFStore) is built for large arrays that don't fit in memory — you can efficiently read a slice of a huge file without loading all of it:

# pip install tables
store = pd.HDFStore("mydata.h5")
store["prices"] = df          # write, dictionary-style
print(store["prices"])        # read back, dictionary-style
store.close()

HDF5 is best for "write-once, read-many" workloads — it isn't a database, and concurrent writers can corrupt the file.

Reading huge files in chunks

For files too large to comfortably fit in memory, chunksize turns read_csv into an iterator — you process the file piece by piece instead of all at once:

import io
 
csv_text = "key,amount\nA,10\nB,20\nA,30\nC,40\nB,50\nA,60\n"
chunker = pd.read_csv(io.StringIO(csv_text), chunksize=2)  # 2 rows per chunk
 
totals = pd.Series([], dtype="int64")
for piece in chunker:
    totals = totals.add(piece.groupby("key")["amount"].sum(), fill_value=0)
 
print(totals.sort_values(ascending=False))

Each piece is a normal, small DataFrame — you aggregate it and merge the running total with .add(..., fill_value=0), so no key is ever lost between chunks.

Pulling JSON from a web API

Many services expose data as a JSON HTTP response. The requests library fetches it; resp.json() parses it into plain Python dicts/lists, which pd.DataFrame can consume directly:

import requests
 
url = "https://api.github.com/repos/pandas-dev/pandas/issues"
resp = requests.get(url)
resp.raise_for_status()  # raise an error early if the request failed
 
data = resp.json()  # list of dicts, one per issue
issues = pd.DataFrame(data, columns=["number", "title", "state"])
print(issues.head())

The exercises below simulate the parsed JSON as a plain Python list of dicts (exactly what resp.json() would hand you), so you can practice the DataFrame-building step without needing live network access.

Common pitfalls

• Parquet and HDF5 both need an extra package installed (pyarrow/fastparquet, tables) — a missing package produces a clear ImportError, not a silent failure.
• Always call resp.raise_for_status() after requests.get() — otherwise a failed request (like a 404) can silently hand you an error page instead of real data.
• When processing chunks, remember each chunk is aggregated independently — you must combine the partial results yourself (as with .add(..., fill_value=0) above).

Visualize it

  flowchart TD
  A["I need to load or save data"] --> B{"Where does it live?"}
  B -->|"Local, short-term cache"| C["pickle
to_pickle / read_pickle"]
  B -->|"Local, long-term / cross-language"| D["Parquet
to_parquet / read_parquet"]
  B -->|"Huge array, partial reads"| E["HDF5
HDFStore"]
  B -->|"Huge CSV, won't fit in memory"| F["chunksize
iterate and aggregate"]
  B -->|"Lives on a remote web service"| G["requests + read_json
resp.json() -> DataFrame"]

🧪 Try It Yourself

Exercise 1 – Round-Trip a DataFrame Through Pickle

Press Run to see what your code prints.

Exercise 2 – Aggregate a File in Chunks

Press Run to see what your code prints.

Exercise 3 – Turn a Parsed JSON Response Into a DataFrame

Press Run to see what your code prints.

Check your understanding

Next

With data loading covered end to end, move on to Correlation and Covariance to start measuring relationships between the columns you've loaded.