A full distribution can be complicated, so we summarize it with a few numbers: the mean (where it’s centered), the variance (how spread out), and the covariance / correlation (how two variables move together). We also formalize independence — when knowing one variable tells you nothing about another. These summaries drive PCA, uncertainty bands, and feature analysis.

A real-life example: height and weight

Collect people’s heights and weights. The means tell you the typical values; the variances tell you how much they spread; and the correlation tells you they rise together (taller people tend to weigh more) — a positive correlation near, say, +0.7. If two features were independent (like height and a random lottery number), their correlation would be ~0 and one would tell you nothing about the other.

Expected value and mean

The expected value of a function $g$ of a random variable is its probability-weighted average:

The mean is the special case $g(x) = x$: $\boldsymbol\mu = \mathbb{E}_X[\mathbf{x}]$. Expectation is linear: $\mathbb{E}[a\,g + b\,h] = a\,\mathbb{E}[g] + b\,\mathbb{E}[h]$. Two other “averages” exist: the median (middle value, robust to outliers) and the mode (most likely value / density peak).

Variance and covariance

The variance measures spread — the expected squared deviation from the mean:

The covariance measures how two variables vary together:

For a vector $\mathbf{x}$, these assemble into the symmetric, positive-semidefinite covariance matrix $\Sigma$, with variances on the diagonal and covariances off it.

Correlation

Covariance depends on scale, so we normalize it into the correlation, always in $[-1, 1]$:

$+1$ = perfectly rising together, $-1$ = perfectly opposed, $0$ = no linear relationship.

Watch correlation tilt the cloud

The same points, re-correlated. As the correlation sweeps from $-1$ to $+1$, the scatter cloud tilts: negative → downward slope, zero → a round blob (no linear relation), positive → upward slope:

Independence

Two random variables are statistically independent iff their joint factorizes:

Then $p(\mathbf{y}\mid\mathbf{x}) = p(\mathbf{y})$ — knowing $\mathbf{x}$ tells you nothing about $\mathbf{y}$. A crucial subtlety:

Conditional independence $X \perp Y \mid Z$ means $p(\mathbf{x}, \mathbf{y} \mid \mathbf{z}) = p(\mathbf{x}\mid\mathbf{z})\,p(\mathbf{y}\mid\mathbf{z})$ — the backbone of graphical models. In ML, data is usually assumed i.i.d. (independent and identically distributed).

  flowchart TD
  D["distribution p(x)"] --> M["mean μ = E[x]"]
  D --> V["variance V[x] = E[x²] − E[x]²"]
  D2["joint p(x, y)"] --> C["covariance Cov[x,y]"]
  C --> R["correlation = Cov / √(V[x]V[y]) ∈ [−1, 1]"]
  D2 --> I["independent iff p(x,y) = p(x)p(y)"]
  I -.-> note["independent ⇒ Cov = 0
(but NOT the converse)"]

NumPy

import numpy as np
rng = np.random.default_rng(0)
 
# correlated height/weight-like data
h = rng.normal(170, 8, 5000)
w = 0.9 * (h - 170) + 65 + rng.normal(0, 4, 5000)   # weight rises with height
 
print("mean height:", round(h.mean(), 2))
print("std  height:", round(h.std(), 2))
print("covariance:\n", np.round(np.cov(h, w), 2))       # 2x2 covariance matrix
print("correlation:", round(np.corrcoef(h, w)[0, 1], 3))  # in [-1, 1]
 
# zero covariance but NOT independent: Y = X^2 with symmetric X
x = rng.normal(0, 1, 100000); y = x**2
print("Cov(X, X²) ≈", round(np.cov(x, y)[0, 1], 3), "(≈0, yet dependent!)")

import numpy as np
rng = np.random.default_rng(0)
 
# correlated height/weight-like data
h = rng.normal(170, 8, 5000)
w = 0.9 * (h - 170) + 65 + rng.normal(0, 4, 5000)   # weight rises with height
 
print("mean height:", round(h.mean(), 2))
print("std  height:", round(h.std(), 2))
print("covariance:\n", np.round(np.cov(h, w), 2))       # 2x2 covariance matrix
print("correlation:", round(np.corrcoef(h, w)[0, 1], 3))  # in [-1, 1]
 
# zero covariance but NOT independent: Y = X^2 with symmetric X
x = rng.normal(0, 1, 100000); y = x**2
print("Cov(X, X²) ≈", round(np.cov(x, y)[0, 1], 3), "(≈0, yet dependent!)")

mean height: 170.04
std  height: 7.98
covariance:
 [[63.7  57.3]
 [57.3 67.5]]
correlation: 0.873
Cov(X, X²) ≈ 0.006 (≈0, yet dependent!)

mean height: 170.04
std  height: 7.98
covariance:
 [[63.7  57.3]
 [57.3 67.5]]
correlation: 0.873
Cov(X, X²) ≈ 0.006 (≈0, yet dependent!)

Why this matters for ML

• Covariance matrices are the input to PCA and Gaussian models; their eigenstructure is the data’s principal directions.
• Correlation guides feature selection and reveals redundant/collinear features.
• Independence assumptions (i.i.d. data, naive Bayes, factorized posteriors) make otherwise intractable models computable.

🧪 Try It Yourself

Exercise 1 – Mean and variance

Exercise 2 – Correlation

Exercise 3 – Independence check

Recap

• Mean (center), variance (spread), and covariance/correlation (joint variation) summarize distributions; expectation is linear.
• The covariance matrix is symmetric positive-semidefinite — the object PCA and Gaussians act on.
• Independence means the joint factorizes; zero covariance does not imply independence (it only rules out linear dependence).

Next: the most important distribution in all of ML — the Gaussian.