Correlation and Covariance

What correlation and covariance answer

Both measure how two variables move together:

• Covariance tells you the direction of the relationship (positive or negative) but its size depends on the scale of your data, which makes it hard to compare across different variables.
• Correlation rescales covariance into a number between -1 and 1, so it’s directly comparable no matter what units the original data was in.

A correlation near 1 means “moves up together,” near -1 means “one goes up as the other goes down,” and near 0 means “no clear linear relationship.”

Series.corr and Series.cov

Correlation and covariance of two Series

import pandas as pd
 
ad_spend = pd.Series([1, 2, 3, 4, 5])
sales = pd.Series([2, 4, 6, 8, 10])  # perfectly proportional to ad_spend
 
print(round(ad_spend.corr(sales), 2))  # 1.0 — perfectly correlated
print(ad_spend.cov(sales))             # 5.0 — positive, but scale-dependent

Both methods automatically use only the overlapping, non-missing, aligned-by-index values from the two Series — you don’t need to clean them up first.

DataFrame.corr and DataFrame.cov: full matrices

Call .corr() (or .cov()) directly on a DataFrame to get every pairwise combination of columns at once:

A full correlation matrix

df = pd.DataFrame({
    "x": [1, 2, 3, 4, 5],
    "y": [2, 4, 6, 8, 10],
    "z": [5, 3, 4, 2, 1],
})
 
print(df.corr())

The diagonal is always 1.0 — every column is perfectly correlated with itself. z here trends opposite to x and y, so its correlation with them is negative.

corrwith: one column against many

corrwith computes the correlation of every column in a DataFrame against a single Series (or another DataFrame’s matching columns) — useful for quickly checking which columns move with a target metric:

corrwith against a target column

print(df.corrwith(df["x"]))

Common pitfalls

• Correlation measures a linear relationship only — two variables can have a strong non-linear pattern (like a U-shape) and still show a correlation near 0.
• A high correlation is not causation — McKinney’s book pairs this topic with data alignment precisely because spurious correlations often come from misaligned data.
• corrwith(axis="columns") compares row-by-row instead of column-by-column — easy to reach for the wrong axis by habit.

Visualize it

diagram From two columns to one number mermaid

  flowchart LR
  A["Series A, Series B
(aligned by index)"] --> B["cov(A, B)
direction, but scale-dependent"]
  B --> C["corr(A, B)
rescaled to -1 .. 1"]
  D["Whole DataFrame"] --> E["df.corr()
matrix of every pair"]

corr rescales covariance into a bounded -1 to 1 score; DataFrame.corr repeats that for every pair of columns to build a matrix.

sketch Scatter plots at different correlation strengths p5.js Pause Restart

As points hug a straight line more tightly, the correlation moves toward +1 or -1; a shapeless cloud sits near 0.

From the book

McKinney demonstrates corr and cov on real stock return data (returns["MSFT"].corr(returns["IBM"])), computed from percent changes, not raw prices. That’s an important habit: raw price series usually trend together simply because both are rising over time, which inflates correlation. Converting to returns (or percent change) first isolates the actual co-movement you care about.

🧪 Try It Yourself

Exercise 1 – Compute Correlation Between Two Series

pch.exerciseTag pch.exerciseTitle

Exercise 2 – Build a Full Correlation Matrix

pch.exerciseTag pch.exerciseTitle

Exercise 3 – Compare Many Columns Against One Target

pch.exerciseTag pch.exerciseTitle

Next

Correlation compares columns side by side. Grouping and Aggregations (groupby, agg) covers summarizing rows instead — split, apply, and combine.