When Models Meet Data Overview

Chapters 2 through 7 built the mathematics: linear algebra, geometry, matrix decompositions, calculus, probability, optimization. Chapter 8 is where that mathematics meets a table of numbers.

The book states its own plan: “there are two different senses in which we use the phrase ‘machine learning algorithm’: training and prediction”, plus “the idea of selecting among different models”. Then it names the three sections that matter — empirical risk minimization (§8.2), maximum likelihood (§8.3), and probabilistic models (§8.4) — a graphical language for them (§8.5), and how to choose among them (§8.6).

The one question

Everything in this chapter is an answer to a single problem:

You can measure how well a model does on the data you have. You care how well it does on data you do not have.

That is the gap between Equation 8.7’s empirical risk and Equation 8.10’s expected risk — one computable, one not. Regularization, priors, posteriors, cross-validation, the marginal likelihood, AIC and BIC are all attempts to say something about the second quantity using only the first.

Three flavours of learning

The chapter gives three answers, and the module measures where they agree and where they do not.

	§8.2 empirical risk	§8.3 likelihood / MAP	§8.4 Bayesian inference
what you write down	a loss	a likelihood	a joint distribution
what you do to it	minimise	maximise	integrate
computational problem	optimization	optimization	integration
the answer	a point	a point	a distribution
needs a prior	no	MLE no, MAP yes	yes

Two of the three turn out to be the same thing. §8.2.3’s penalty and §8.3.2’s prior produce identical estimates — measured to $1.24\times10^{-14}$ across six orders of magnitude — with $\lambda = \sigma^2/(N\tau^2)$. The third is genuinely different, and the difference is measurable: a nominal $95\%$ interval built from the full posterior achieves $0.9498$ coverage in extrapolation where a plug-in interval achieves $0.1796$.

§8.1 Data, models and learning

Three components, each with a section of its own.

Data as vectors. Data is assumed tabular — rows are examples, columns are features — and in “a tidy format”. Getting there involves real choices: a categorical column can become $\{0, 1\}$ or $\{-1, +1\}$; a postcode is “not just a string of characters but actually encodes an area in London”. The book is explicit that it does not discuss feature construction, which “depend[s] on domain expertise and require[s] careful engineering”.

Models. Two readings, and the chapter uses both. A model as a function — a predictor mapping inputs to outputs. A model as a probabilistic model — a distribution over what the data-generating process might produce. The second costs more and buys the thing the first cannot express: a statement about how confident the prediction is.

Learning. Three phases, and confusing them is the chapter’s central error mode:

phase	what happens	what it must not touch
prediction / inference	run the fitted model on new inputs	—
training / parameter estimation	fit $\boldsymbol\theta$	the test set
hyperparameter tuning / model selection	choose $\lambda$, the degree, the architecture	the set you will report on

The third row is why §8.6.1 nests two loops, and the cost of ignoring it is measurable: flat cross-validation reports $13.5\%$ better than chance on data containing no signal at all.

The pages

#	page	book section	the measured claim
801	Data, Models and Learning	§8.1	the three phases, and what each may see
802	Empirical Risk Minimization	§8.2	Eq 8.7 against Eq 8.10, and the loss you choose changes the winner
803	Regularization and Cross-Validation	§8.2.3–8.2.4	the penalty that trades fit for $\lVert\boldsymbol\theta\rVert$, and how to pick it
804	Maximum Likelihood Estimation	§8.3.1	the NLL is affine in the empirical risk — same minimiser, $1.8\times10^{-15}$
805	MAP Estimation and Model Fitting	§8.3.2–8.3.4	a prior is the penalty: $\lambda = \sigma^2/(N\tau^2)$, to $1.24\times10^{-14}$
806	Probabilistic Modeling and Inference	§8.4	Eq 8.23 holds $0.95$ coverage where the plug-in falls to $0.1796$
807	Directed Graphical Models	§8.5	Example 8.9’s four d-separation claims, four for four
808	Model Selection	§8.6	the automatic Occam’s razor, and the Jeffreys–Lindley paradox measured
809	Chapter 8 Worked Problems	—	eight problems, this module’s own
810	Chapter 8 Formula Sheet	—	every equation and every measured constant

Chapter 8 has no end-of-chapter exercises

Unlike Chapters 2 through 7, this chapter runs from §8.1 straight to §8.6.4 and then into Chapter 9. Page 809’s problems are this module’s own, built from claims the chapter makes, and should not be attributed to Deisenroth, Faisal and Ong.

Four claims worth knowing before you start

1. Overfitting has the lowest training risk. Measured on Figure 8.8’s three cases: the overfitting model scores $0.005232$ on training data — $22\times$ better than the model that actually generalises — and $3.9173$ on held-out data, a ratio of $748.8$. Underfitting announces itself; overfitting does not. The only training-side warning is the parameter norm, $3616.98$ against $5.67$.

2. A regularization penalty is a prior, exactly. Not analogously — identically. The largest disagreement between ridge regression and MAP across $120$ prior widths is $1.24\times10^{-14}$. What the probabilistic vocabulary buys is not a different answer but a meaning for $\lambda$: a ratio of noise variance to prior variance, which immediately predicts that noisier data should be shrunk harder and more data less.

3. A point estimate cannot say “I don’t know.” Its predictive interval has width $2 \times 1.96 \times \sigma$ everywhere, because $\sigma$ contains no reference to where you asked. Measured, that interval covers the truth $17.96\%$ of the time in far extrapolation while labelled $95\%$. The full posterior covers $94.98\%$.

4. Every model-selection number is hostage to something. Cross-validation is hostage to how hard you searched — $13.5\%$ better than chance on pure noise. The Bayes factor is hostage to the prior — with the data held fixed, its verdict flips from “decisively degree 9” to “decisively degree 1” as $\tau^2$ widens past $3.71\times10^{6}$. There is no criterion that is free.

Where it goes next

The book’s closing line for the chapter: the rest of this part shows “how the three different flavors of learning in Sections 8.2, 8.3, and 8.4 are applied to the four pillars of machine learning”.

pillar	chapter	which flavour dominates
regression	9	all three — and the conjugate evidence §8.6.3 promised
dimensionality reduction	10	latent variables, §8.4.3, and Eq 8.28
density estimation	11	mixtures, EM, and Eq 8.25’s marginalisation
classification	12	empirical risk with a different loss, §8.2

So Chapter 8 is not a detour before the applications. It is the vocabulary the remaining four chapters are written in — and Equation 8.28, the latent posterior given the parameters, is computed explicitly in both Chapter 10 and Chapter 11.

Recall card

• The chapter answers one question: you can measure performance on the data you have, and you care about the data you do not have. That is the gap between Equation 8.7’s empirical risk and Equation 8.10’s expected risk.
• Three flavours of learning: Section 8.2 minimises a loss, Section 8.3 maximises a likelihood, Section 8.4 integrates over a joint distribution. The first two solve an optimization problem and return a point; the third solves an integration problem and returns a distribution.
• Two of the three are the same estimator. A squared-norm penalty and a zero-mean Gaussian prior give identical answers with lambda equal to sigma squared over N tau squared, measured to 1.24e-14.
• Section 8.1’s three components: data as vectors, models as functions or as probability distributions, and learning as the search for parameters that generalise.
• Three phases of learning, and confusing them is the chapter’s central error: prediction, training, and hyperparameter tuning. The phase that chooses must not be the phase that reports.
• Overfitting has the lowest training risk of the three cases, and its expected risk is 748.8 times its training risk. Underfitting announces itself from training data alone; overfitting does not.
• A point estimate cannot express ignorance. Its interval has constant width everywhere, and measured, it covers the truth 17.96 percent of the time in far extrapolation while labelled 95 percent. The full posterior covers 94.98 percent.
• Every model-selection criterion is hostage to something — cross-validation to how hard you searched, the Bayes factor to the prior width. There is no free one.
• Section 8.5’s graphical language describes the joint distribution, and d-separation reads conditional independence off the picture without algebra.
• Chapter 8 is the vocabulary for Chapters 9 through 12 — regression, dimensionality reduction, density estimation and classification, the four pillars.

Start here: Data, Models and Learning