Beyond CSV: faster and richer formats

CSV is human-readable but slow to parse and loses type information (everything comes back as text until pandas guesses otherwise). Once a project outgrows quick CSV exports, three tools solve different problems:

• Binary formats (picklepickle, Parquet, HDF5) — faster reads/writes, types preserved.
• Chunked reading (chunksizechunksize) — process files bigger than memory, piece by piece.
• Web APIs (requestsrequests + JSON) — pull data directly from a live service instead of a file at all.

Pickle: quick, Python-only serialization

to_pickleto_pickle / read_pickleread_pickle dump a pandas object to disk (or any file-like object) almost instantly, preserving dtypes exactly:

import pandas as pd
import io
 
df = pd.DataFrame({"a": [1, 2, 3], "b": ["x", "y", "z"]})
 
buffer = io.BytesIO()
df.to_pickle(buffer)   # in real code: df.to_pickle("data.pkl")
 
buffer.seek(0)
restored = pd.read_pickle(buffer)   # in real code: pd.read_pickle("data.pkl")
print(restored)

import pandas as pd
import io
 
df = pd.DataFrame({"a": [1, 2, 3], "b": ["x", "y", "z"]})
 
buffer = io.BytesIO()
df.to_pickle(buffer)   # in real code: df.to_pickle("data.pkl")
 
buffer.seek(0)
restored = pd.read_pickle(buffer)   # in real code: pd.read_pickle("data.pkl")
print(restored)

Parquet and Feather: columnar, cross-language, fast

Apache Parquet (via to_parquetto_parquet / read_parquetread_parquet) stores data column-by-column with compression, keeping full type information — and unlike pickle, it’s readable from other languages (R, Spark, Java) too:

# pip install pyarrow  (or: conda install pyarrow)
df.to_parquet("data.parquet")
loaded = pd.read_parquet("data.parquet")
print(loaded.dtypes)

# pip install pyarrow  (or: conda install pyarrow)
df.to_parquet("data.parquet")
loaded = pd.read_parquet("data.parquet")
print(loaded.dtypes)

Feather (to_featherto_feather / read_featherread_feather) is a similar columnar format optimized for very fast reads/writes, most useful for short-lived caches between processing steps.

HDF5: many datasets in one file

HDF5 (via pandas.HDFStorepandas.HDFStore) is built for large arrays that don’t fit in memory — you can efficiently read a slice of a huge file without loading all of it:

# pip install tables
store = pd.HDFStore("mydata.h5")
store["prices"] = df          # write, dictionary-style
print(store["prices"])        # read back, dictionary-style
store.close()

# pip install tables
store = pd.HDFStore("mydata.h5")
store["prices"] = df          # write, dictionary-style
print(store["prices"])        # read back, dictionary-style
store.close()

HDF5 is best for “write-once, read-many” workloads — it isn’t a database, and concurrent writers can corrupt the file.

Reading huge files in chunks

For files too large to comfortably fit in memory, chunksizechunksize turns read_csvread_csv into an iterator — you process the file piece by piece instead of all at once:

import io
 
csv_text = "key,amount\nA,10\nB,20\nA,30\nC,40\nB,50\nA,60\n"
chunker = pd.read_csv(io.StringIO(csv_text), chunksize=2)  # 2 rows per chunk
 
totals = pd.Series([], dtype="int64")
for piece in chunker:
    totals = totals.add(piece.groupby("key")["amount"].sum(), fill_value=0)
 
print(totals.sort_values(ascending=False))

import io
 
csv_text = "key,amount\nA,10\nB,20\nA,30\nC,40\nB,50\nA,60\n"
chunker = pd.read_csv(io.StringIO(csv_text), chunksize=2)  # 2 rows per chunk
 
totals = pd.Series([], dtype="int64")
for piece in chunker:
    totals = totals.add(piece.groupby("key")["amount"].sum(), fill_value=0)
 
print(totals.sort_values(ascending=False))

Each piecepiece is a normal, small DataFrame — you aggregate it and merge the running total with .add(..., fill_value=0).add(..., fill_value=0), so no key is ever lost between chunks.

Pulling JSON from a web API

Many services expose data as a JSON HTTP response. The requestsrequests library fetches it; resp.json()resp.json() parses it into plain Python dicts/lists, which pd.DataFramepd.DataFrame can consume directly:

import requests
 
url = "https://api.github.com/repos/pandas-dev/pandas/issues"
resp = requests.get(url)
resp.raise_for_status()  # raise an error early if the request failed
 
data = resp.json()  # list of dicts, one per issue
issues = pd.DataFrame(data, columns=["number", "title", "state"])
print(issues.head())

import requests
 
url = "https://api.github.com/repos/pandas-dev/pandas/issues"
resp = requests.get(url)
resp.raise_for_status()  # raise an error early if the request failed
 
data = resp.json()  # list of dicts, one per issue
issues = pd.DataFrame(data, columns=["number", "title", "state"])
print(issues.head())

The exercises below simulate the parsed JSON as a plain Python list of dicts (exactly what resp.json()resp.json() would hand you), so you can practice the DataFrameDataFrame-building step without needing live network access.

Common pitfalls

• Parquet and HDF5 both need an extra package installed (pyarrowpyarrow/fastparquetfastparquet, tablestables) — a missing package produces a clear ImportErrorImportError, not a silent failure.
• Always call resp.raise_for_status()resp.raise_for_status() after requests.get()requests.get() — otherwise a failed request (like a 404) can silently hand you an error page instead of real data.
• When processing chunks, remember each chunk is aggregated independently — you must combine the partial results yourself (as with .add(..., fill_value=0).add(..., fill_value=0) above).

Visualize it

  flowchart TD
  A["I need to load or save data"] --> B{"Where does it live?"}
  B -->|"Local, short-term cache"| C["pickle
to_pickle / read_pickle"]
  B -->|"Local, long-term / cross-language"| D["Parquet
to_parquet / read_parquet"]
  B -->|"Huge array, partial reads"| E["HDF5
HDFStore"]
  B -->|"Huge CSV, won't fit in memory"| F["chunksize
iterate and aggregate"]
  B -->|"Lives on a remote web service"| G["requests + read_json
resp.json() -> DataFrame"]

🧪 Try It Yourself

Exercise 1 – Round-Trip a DataFrame Through Pickle

Exercise 2 – Aggregate a File in Chunks

Exercise 3 – Turn a Parsed JSON Response Into a DataFrame

Next

With data loading covered end to end, move on to Correlation and Covariance to start measuring relationships between the columns you’ve loaded.