Correlation vs Causation (Pearson, Spearman)

Correlation measures association

Correlation answers:

• “Do variables move together?”

It does not answer:

• “Does X cause Y?”

Pearson correlation

• Measures linear relationship
• Sensitive to outliers

Pearson

import numpy as np
from scipy import stats
 
x = np.array([1, 2, 3, 4, 5, 6])
y = np.array([2, 4, 5, 4, 5, 7])
 
r, p = stats.pearsonr(x, y)
print("r:", r)
print("p:", p)

Spearman correlation

• Uses ranks
• Captures monotonic relationships
• More robust to outliers and non-linearity

Spearman

import numpy as np
from scipy import stats
 
x = np.array([1, 2, 3, 4, 5, 6])
y = np.array([10, 9, 7, 6, 3, 1])
 
rho, p = stats.spearmanr(x, y)
print("rho:", rho)
print("p:", p)

Visualize it

Pearson’s r measures how tightly two variables move together: near +1 the points hug a rising line, near 0 there’s no pattern, near −1 they hug a falling line. Watch the cloud tighten and flip as r changes:

sketch Correlation, from strong to none p5.js Pause Restart

r near +1 means points follow a rising line, 0 means no linear pattern, -1 means a falling line.

Practical guidance

• Plot scatter first.
• Consider transformations (log) if scales vary.
• Be cautious: confounders can create spurious correlation.

Why correlation isn’t causation

diagram Correlation can hide a confounder mermaid

  flowchart LR
  Z["Confounder
(e.g. summer heat)"] --> X["Ice cream sales"]
  Z --> Y["Drowning incidents"]
  X -.->|"correlated, not causal"| Y

Two variables can move together purely because a third variable drives both.

From the book

DataFrame.corr() computes Pearson correlation across every numeric column pair in one call — handy for a quick scan, but McKinney’s own examples (like the cov/corrcoef pair in NumPy) are careful to only describe linear association. Nothing in the correlation matrix itself can tell you which variable, if any, is causing the other — that requires domain knowledge or a designed experiment (see A/B Testing Basics).

🧪 Try It Yourself

Exercise 1 – Pearson correlation

pch.exerciseTag pch.exerciseTitle

Exercise 2 – Spearman for a monotonic (non-linear) trend

pch.exerciseTag pch.exerciseTitle

Exercise 3 – Spotting a possible confounder

pch.exerciseTag pch.exerciseTitle

Next

Continue to Non-Parametric Tests for association and comparison tests that don’t assume normal, linear relationships.