Linear Regression Overview

Chapter 8 built a vocabulary and named three flavours of learning. Chapter 9 applies all three to one problem, in order, and the problem is the first of the book’s four pillars: “we aim to find a function $f$ that maps inputs $\mathbf{x} \in \mathbb{R}^D$ to corresponding function values $f(\mathbf{x}) \in \mathbb{R}$.”

The book’s framing: “we will apply the mathematical concepts from Chapters 2, 5, 6, and 7 to solve linear regression (curve fitting) problems” — and the task is “to find a function that not only models the training data, but generalizes well.”

One problem, three times

	§9.2.1 maximum likelihood	§9.2.3 MAP	§9.3 Bayesian
what you write down	a likelihood	likelihood $\times$ prior	the same, and you integrate
the answer	a point	a point	a distribution
exists when $K > N$	no	yes	yes
predictive interval	width $\sigma$, everywhere	width $\sigma$, everywhere	widens off the data
measured coverage	$0.7419$ / $0.1112$	the same	$\mathbf{0.9487}$ / $\mathbf{0.9499}$

Two of the three give the same prediction curve. Measured to $8.4\times10^{-14}$: MAP and the Bayesian posterior mean are one curve, because for a Gaussian the mode is the mean. So the case for §9.3 is not about the line — it is entirely about the shading, and the shading is what turns $0.1112$ into $0.9499$.

Why this chapter is where the measuring pays off

Everything here has a closed form, so every claim can be checked exactly rather than approximately. Four things the chapter states, and what measuring them turned up:

The book gives two different $\lambda$ for one correspondence. Equation 9.33’s text says $\lambda = 1/(2b^2)$ and Equation 9.34’s says $\lambda = \sigma^2/b^2$. Measured across five settings, only the second reproduces Equation 9.31 — and Chapter 8 supplies a third, $\sigma^2/(N\tau^2)$, for its own normalisation. All three are internally correct. A $\lambda$ is meaningless without the objective it belongs to.

Equation 9.22’s noise estimate is biased by exactly $(N-K)/N$. The book states it without caveat. Measured at $N = 10$, $K = 5$ — Example 9.5’s own setting — it returns half the true variance, and §9.4’s geometry explains why: $\mathrm{tr}(\mathbf{I}-P) = N-K$, so the residual never occupies all $N$ dimensions.

Example 9.7’s prior is far wider than its figure can show. With the book’s own $\mathcal{N}(\mathbf{0}, \tfrac14\mathbf{I})$ and degree-5 monomials, the $95\%$ band is $3125.59$ units wide at $x = 5$ — on axes drawn from $-4$ to $4$. It fits inside them at $48$ of the $200$ plotted inputs. The shape the figure draws is right; the scale is the part worth measuring.

Equation 9.64 is a derivation, not an implementation. It inverts an $N\times N$ matrix for a model with $K$ parameters. Woodbury gives the same number through a $K\times K$ one — $4341\times$ faster at $N = 4000$ — and its determinant factor is $\mathbf{S}_N^{-1}$, already computed by Theorem 9.1.

The pages

#	page	book section	the measured claim
901	Problem Formulation	§9.1–9.2	the likelihood integrates to $1/\lvert x\rvert$ over $\boldsymbol\theta$
902	Maximum Likelihood Estimation for Linear Regression	§9.2.1	Eq 9.22 is biased by exactly $(N-K)/N$
903	Overfitting in Linear Regression	§9.2.2	the training error never rises — nine steps, zero increases
904	MAP Estimation and Regularization	§9.2.3–9.2.4	only $\lambda = \sigma^2/b^2$ reproduces Eq 9.31
905	Bayesian Linear Regression	§9.3.1–9.3.2	the prior’s $95\%$ band is $3125.59$ wide at $x = 5$
906	The Parameter Posterior	§9.3.3	precisions add, so the update runs online to $4.1\times10^{-12}$
907	Posterior Predictions	§9.3.4	$\mathbf{S}_N$ is unchanged by shuffling the targets
908	Computing the Marginal Likelihood	§9.3.5	the evidence picks the same degree a test set does
909	Maximum Likelihood as Orthogonal Projection	§9.4	$\mathrm{tr}(P) = K$, which explains page 902’s bias
910	Chapter 9 Worked Problems	—	eight problems, this module’s own
911	Chapter 9 Formula Sheet	—	every equation and every measured constant

Chapter 9 has no end-of-chapter exercises

Like Chapter 8, it runs straight from §9.5 into Chapter 10. Page 910’s problems are this module’s own, built from claims the chapter makes, and should not be attributed to Deisenroth, Faisal and Ong.

The thread that runs through it

One number appears on three pages by three different routes, and noticing that is most of understanding Bayes’ theorem in this setting:

where it appears	page	value
the constant offset making Theorem 9.1 consistent	906	$+30.453407$
the training targets’ density under the prior predictive	907	$-30.453407$
Equation 9.64 evaluated directly	908	$-30.453407$

It is $\log p(\mathcal{Y}\mid\mathcal{X})$ — the normaliser page 901 found missing when the likelihood turned out to integrate to $1/\lvert x\rvert$ rather than $1$. Maximum likelihood can ignore it because an argmax ignores scale. Everything else in the chapter needs it, and §8.6 needs it to compare models at all.

What is actually new here

Chapter 8 established the ideas. This chapter supplies the closed forms — and three results that only appear once you have them:

Precisions add, so learning can be sequential. Equation 9.43b is a sum over observations, so feeding the data in any order or grouping gives the same posterior. Measured over $500$ points fed one at a time: $6.8\times10^{-15}$ from the batch answer, in $O(K^2)$ memory, with the design matrix never assembled. The book computes it in one shot; nothing requires that.

The error bars are known before the experiment. $\mathbf{S}_N$ contains no $\mathbf{y}$ — measured, shuffling the targets changes it by exactly zero. So you can choose where to measure to minimise the interval where it matters: a clustered design gives a worst $95\%$ half-width of $286.95$ against $0.474$ for a spread one, a factor of $605$, decided in advance. The same property is why the shading does not widen when the model fits badly.

The estimator is a projection, and that explains two earlier puzzles. $\mathrm{tr}(P) = K$ gives Equation 9.22’s bias exactly; and since a projection depends on the subspace rather than the basis, page 902’s conditioning catastrophe ($\kappa = 2.2\times10^{26}$ for monomials against $1.000000$ for an orthonormal basis, same fit to $4.8\times10^{-13}$) was always a coordinate problem.

Where it goes next

§9.5 points three ways, and all three are visible in what the chapter already computed:

pointer	and where it already appeared
generalized linear models — Eq 9.72, a deep network as a composition of them	Chapter 8’s §8.3.4, and the note that maximum likelihood becomes non-convex
Gaussian processes — a distribution placed directly on functions	page 905 measured the induced kernel $\phi(x_a)^\top\mathbf{S}_0\phi(x_b)$, including a $-204.75$
sparsity priors — Laplace, i.e. LASSO	page 904 measured ridge producing zero exact zeros at any $\lambda$ against LASSO’s $27$ of $30$

And the remaining pillars: Chapter 10 applies the same machinery to dimensionality reduction, Chapter 11 to density estimation, Chapter 12 to classification.

Recall card

• Chapter 9 solves one problem three times — maximum likelihood, MAP, and full Bayesian inference — applying Chapter 8’s three flavours of learning to the first of the four pillars.
• Two of the three give the same prediction curve, to 8.4e-14, because for a Gaussian the mode is the mean. The case for Section 9.3 is entirely about the interval.
• And the interval is what turns 0.1112 into 0.9499 in measured coverage just outside the data.
• The book names two different lambdas for one correspondence, and Chapter 8 supplies a third. All are internally correct; a lambda is meaningless without its objective.
• Equation 9.22’s noise estimate is biased by exactly (N-K)/N, returning half the truth in Example 9.5’s own setting. Section 9.4 explains why: the trace of I minus P is N minus K.
• Example 9.7’s prior implies a 95 percent band 3125.59 units wide at x = 5, on a figure whose axes run from minus four to four.
• Equation 9.64 inverts an N-by-N matrix for a K-parameter model. Woodbury gives the same number 4341 times faster, reusing the posterior precision Theorem 9.1 already computed.
• One number appears on three pages by three routes: 30.453407, the log marginal likelihood — the normaliser page 901 found missing.
• Precisions add, so the posterior update runs online in fixed memory, matching the batch answer to 6.8e-15 over 500 points.
• The error bars never see the targets, so they can be minimised during experimental design — a factor of 605 between a clustered and a spread layout — and they cannot warn you that the model is wrong.
• The estimator is an orthogonal projection, which explains both the noise-variance bias and why the monomial basis’s conditioning disaster was avoidable.
• Section 9.5 points to generalized linear models, Gaussian processes and sparsity priors, and the chapter has already computed a piece of each.

Start here: Problem Formulation