All of probability reduces to two rules: the sum rule (to remove a variable) and the product rule (to split a joint into pieces). From these follows Bayes’ theorem — the formula that inverts a likelihood into a posterior belief. Bayes is the engine of spam filters, medical diagnosis, and every Bayesian machine-learning method.

A real-life example: a scary-looking positive test

A test for a rare disease is “99% accurate,” and you test positive. Should you panic? Surprisingly, no — if the disease is rare, most positives are false positives. Bayes’ theorem combines the test result (the likelihood) with the disease’s rarity (the prior) to give the real chance you’re sick (the posterior), which can be far below the “99%” your intuition screams. This base-rate reasoning is Bayes in action, and it’s where human intuition famously fails.

The two rules

For random variables $\mathbf{x}, \mathbf{y}$ with joint $p(\mathbf{x}, \mathbf{y})$:

Sum rule (marginalization) — remove a variable by summing/integrating it out:

Product rule — factor a joint into a conditional times a marginal:

That’s it. Every other identity in probability is built from these two.

Bayes’ theorem

Since the product rule can factor either way ($p(\mathbf{x},\mathbf{y}) = p(\mathbf{y}\mid\mathbf{x})p(\mathbf{x}) = p(\mathbf{x}\mid\mathbf{y})p(\mathbf{y})$), rearranging gives Bayes’ theorem:

The four pieces:

• Prior $p(\mathbf{x})$ — what you believed before seeing data.
• Likelihood $p(\mathbf{y} \mid \mathbf{x})$ — how probable the data is under each hypothesis.
• Posterior $p(\mathbf{x} \mid \mathbf{y})$ — your updated belief after the data.
• Evidence $p(\mathbf{y}) = \int p(\mathbf{y}\mid\mathbf{x})p(\mathbf{x})\,d\mathbf{x}$ — a normalizer ensuring the posterior sums to 1.

See base rates in action

Here’s a population of 400 people. A disease affects 5% (blue). A test with 90% sensitivity and 90% specificity flags the ringed people as positive. Look at the ringed group: most are actually healthy (false positives), because healthy people vastly outnumber sick ones. The posterior $P(\text{sick} \mid +)$ is only ~32%, not 90%:

The test didn’t lie — it just can’t overcome how rare the disease is. That’s why doctors retest, and why ignoring the prior (the base rate) is a classic reasoning error.

The pieces, connected

  flowchart LR
  PRIOR["prior p(x)
belief before data"] --> BAYES["Bayes' theorem"]
  LIK["likelihood p(y|x)
data under hypothesis"] --> BAYES
  BAYES --> POST["posterior p(x|y)
updated belief"]
  EV["evidence p(y)
= Σ p(y|x)p(x) (sum rule)"] --> BAYES
  POST -.-> NEXT["becomes the prior for the next observation"]

NumPy

import numpy as np
 
# Disease base rate and test accuracy
prior_sick = 0.05           # P(sick)
sensitivity = 0.90          # P(+ | sick)
specificity = 0.90          # P(- | healthy)  ->  P(+ | healthy) = 0.10
 
# Evidence via the sum rule: P(+) = P(+|sick)P(sick) + P(+|healthy)P(healthy)
p_pos = sensitivity * prior_sick + (1 - specificity) * (1 - prior_sick)
 
# Bayes: posterior P(sick | +)
posterior = (sensitivity * prior_sick) / p_pos
print(f"P(+)              = {p_pos:.3f}")
print(f"P(sick | +)       = {posterior:.3f}")   # ~0.321
print(f"prior was {prior_sick} -> posterior {posterior:.2f}")

import numpy as np
 
# Disease base rate and test accuracy
prior_sick = 0.05           # P(sick)
sensitivity = 0.90          # P(+ | sick)
specificity = 0.90          # P(- | healthy)  ->  P(+ | healthy) = 0.10
 
# Evidence via the sum rule: P(+) = P(+|sick)P(sick) + P(+|healthy)P(healthy)
p_pos = sensitivity * prior_sick + (1 - specificity) * (1 - prior_sick)
 
# Bayes: posterior P(sick | +)
posterior = (sensitivity * prior_sick) / p_pos
print(f"P(+)              = {p_pos:.3f}")
print(f"P(sick | +)       = {posterior:.3f}")   # ~0.321
print(f"prior was {prior_sick} -> posterior {posterior:.2f}")

P(+)              = 0.140
P(sick | +)       = 0.321
prior was 0.05 -> posterior 0.32

P(+)              = 0.140
P(sick | +)       = 0.321
prior was 0.05 -> posterior 0.32

Why this matters for ML

• Bayesian inference is Bayes’ theorem applied to model parameters: prior belief + data likelihood → posterior over parameters (Chapters 9, 11).
• Naive Bayes classifiers apply Bayes with a conditional-independence assumption — still a strong spam-filter baseline.
• The evidence $p(\mathbf{y})$ powers model selection (Bayesian model comparison), and its intractability motivates variational inference and MCMC.

🧪 Try It Yourself

Exercise 1 – The sum rule (marginalize)

Exercise 2 – Bayes’ theorem

Exercise 3 – Product rule

Recap

• All of probability follows from the sum rule (marginalize out a variable) and the product rule (factor a joint).
• Bayes’ theorem inverts a likelihood into a posterior: posterior ∝ likelihood × prior, normalized by the evidence.
• Base rates matter: a positive test on a rare condition is mostly false positives — ignoring the prior is a classic error.

Next: numbers that summarize distributions, and when variables carry independent information — Summary Statistics and Independence.