Chapter 9 Worked Problems

#	problem	sections	the answer in one line
1	Which $\lambda$ reproduces the MAP estimate?	§9.2.3–9.2.4	two of the three work, with different values
2	Fit 500 points without ever storing them	§9.3.3	batch answer matched to $6.8\times10^{-15}$
3	Do the evidence and cross-validation agree?	§9.3.5, §8.6	both pick $M = 4$ , as does the test set
4	Choose where to measure, before measuring	§9.3.4	a factor of $\mathbf{605}$ in the worst interval
5	How much wider is the honest interval?	§9.3.4	$1.14\times$ inside the data, $19.78\times$ outside
6	Is the projection special to polynomials?	§9.4	no — Gaussian bumps behave identically
7	Fix Equation 9.22	§9.2.1, §9.4	divide by $N-K$ : $0.04017$ against $0.04000$
8	What does a prior do to a singular design?	§9.2.3, §9.3.3	splits tied weight evenly, to $0.000\times10^{0}$

All problems share one setup:

setup.py

import numpy as np
 
SIG = 0.2
 
def truth(x):
    return -np.sin(x / 5) + np.cos(x)
 
def design(x, M):
    return np.vander(np.asarray(x, float), M + 1, increasing=True)
 
def posterior(P, yv, m0, S0, sig=SIG):
    """Theorem 9.1."""
    SN = np.linalg.inv(np.linalg.inv(S0) + P.T @ P / sig ** 2)
    mN = SN @ (np.linalg.solve(S0, m0) + P.T @ yv / sig ** 2)
    return mN, SN
 
def log_ev(P, yv, m0, S0, sig=SIG):
    """Equation 9.64."""
    N = len(yv)
    C = P @ S0 @ P.T + sig ** 2 * np.eye(N)
    d = yv - P @ m0
    _, ld = np.linalg.slogdet(C)
    return float(-0.5 * (d @ np.linalg.solve(C, d) + ld + N*np.log(2*np.pi)))
 
def rmse(a, b):
    return float(np.sqrt(np.mean((a - b) ** 2)))

Problem 1 — Which lambda reproduces the MAP estimate?

Statement. Page 904 found the book naming two different $\lambda$ in adjacent paragraphs, and page 805 a third from Chapter 8. Settle it: given $\sigma^2$ and $b^2$ , which penalised least-squares objective, with which $\lambda$ , returns exactly Equation 9.31’s estimate?

Method. Compute $\boldsymbol\theta_{\text{MAP}}$ , then solve each candidate objective and compare.

problem_1.py

rng = np.random.default_rng(4)
x = np.sort(rng.uniform(-5, 5, 10))
y = truth(x) + SIG * rng.standard_normal(10)
P = design(x, 4)
K, N = P.shape[1], len(y)
b2 = 0.4
th_map = np.linalg.solve(P.T @ P + (SIG**2 / b2) * np.eye(K), P.T @ y)
 
print(f"{'objective':>42} {'lambda':>12} {'max |theta - theta_MAP|':>26}")
cands = [
    ("||y-Phi t||^2 + lam ||t||^2   (Eq 9.32)", SIG**2 / b2, 1.0),
    ("   ... with lam = 1/(2 b^2)  (Eq 9.33)", 1.0/(2*b2), 1.0),
    ("(1/N)||y-Phi t||^2 + lam ||t||^2 (Eq 8.12)", SIG**2/(N*b2), 1.0/N),
]
for name, lam, scale in cands:
    A = scale * (P.T @ P) + lam * np.eye(K)
    t = np.linalg.solve(A, scale * (P.T @ y))
    print(f"{name:>42} {lam:>12.6f} {np.abs(t - th_map).max():>26.3e}")

text

                                 objective       lambda    max |theta - theta_MAP|
   ||y-Phi t||^2 + lam ||t||^2   (Eq 9.32)     0.100000                  0.000e+00
       ... with lam = 1/(2 b^2)  (Eq 9.33)     1.250000                  2.388e-01
(1/N)||y-Phi t||^2 + lam ||t||^2 (Eq 8.12)     0.010000                  7.772e-15

Answer. Two of the three work, with $\lambda$ differing by a factor of $N$ .

objective	$\lambda$	reproduces Eq 9.31?
Eq 9.32, un-normalised data-fit	$\sigma^2/b^2 = 0.1$	yes, exactly
Eq 8.12, data-fit divided by $N$	$\sigma^2/(Nb^2) = 0.01$	yes, to $7.8\times10^{-15}$
Eq 9.33’s value	$1/(2b^2) = 1.25$	no — off by $0.2388$

$\lambda$ is not a property of the model. It is a property of the pair (objective, prior), and reporting one without the other says nothing. Equation 9.33’s statement is true about the two penalty terms and false about the two estimators — the missing factor is $2\sigma^2$ , exactly as page 904 measured.

Problem 2 — Fit 500 points without ever storing them

Statement. §9.3.3 computes the posterior from a full design matrix. Page 906 measured that the update is sequential. Push it: process a stream of 500 observations one at a time, never forming a 500-row matrix, and check the result against the batch computation.

Method. Carry $(\mathbf{m}, \mathbf{S})$ and call Theorem 9.1 with a single-row design matrix each step, feeding the previous posterior in as the prior.

problem_2.py

m0, S0 = np.zeros(5), 0.4 * np.eye(5)
r2 = np.random.default_rng(91)
xs = r2.uniform(-5, 5, 500)
ys = truth(xs) + SIG * r2.standard_normal(500)
Pall = design(xs, 4)
m_batch, S_batch = posterior(Pall, ys, m0, S0)
 
m_seq, S_seq = m0.copy(), S0.copy()
print(f"{'points seen':>13} {'max post. sd':>14} {'||m_N||':>11} "
      f"{'gap to the batch answer':>26}")
for n in range(500):
    m_seq, S_seq = posterior(Pall[n:n+1], ys[n:n+1], m_seq, S_seq)
    if n + 1 in (1, 5, 25, 100, 500):
        print(f"{n+1:>13} {np.sqrt(np.diag(S_seq)).max():>14.6e} "
              f"{np.linalg.norm(m_seq):>11.6f} "
              f"{np.abs(m_seq - m_batch).max():>26.3e}")

text

  points seen   max post. sd     ||m_N||    gap to the batch answer
            1   6.319472e-01    0.004703                  7.992e-01
            5   6.089182e-01    0.224773                  6.626e-01
           25   8.200601e-02    0.893537                  1.803e-01
          100   3.828706e-02    0.881152                  4.778e-02
          500   1.744744e-02    0.875882                  6.772e-15

Answer. $500$ rank-one updates reach the batch answer to $6.772\times10^{-15}$ , using $O(K^2)$ memory throughout.

The middle column is the interesting one. After one observation almost nothing has happened — the largest posterior sd is $0.632$ against a prior $\sqrt{0.4} = 0.632$ . One point cannot constrain five parameters, and what keeps the answer finite is entirely the prior.

By $25$ points the sd has fallen to $0.082$ ; by $500$ to $0.0174$ . That is the $1/\sqrt{N}$ contraction page 906 measured, watched happening.

The “gap” column is not an error. It is the distance from a partial answer to the full one, and it reaching $10^{-15}$ at $n = 500$ is the point: the stream has consumed exactly the same information the batch did, in any order, with no matrix ever assembled.

Problem 3 — Do the evidence and cross-validation agree?

Statement. §9.3.5 selects a degree with no held-out data; §8.6.1 selects one with held-out data. Do they agree?

Method. On the same ten points, compute the log evidence, a 5-fold cross-validated RMSE using the posterior mean as the predictor, and the RMSE on a 200-point test set.

problem_3.py

K5 = 5
idx = np.random.default_rng(3).permutation(len(y))
folds = np.array_split(idx, K5)
xte = np.linspace(-5, 5, 200)
yte = truth(xte) + SIG * np.random.default_rng(1234).standard_normal(200)
 
print(f"{'M':>4} {'log p(Y | X)':>15} {'5-fold CV RMSE':>16} "
      f"{'test RMSE':>12}")
evs, cvs, tes = [], [], []
for M in range(9):
    Pm = design(x, M)
    mm, SS = np.zeros(M+1), 0.4 * np.eye(M+1)
    evs.append(log_ev(Pm, y, mm, SS))
    errs = []
    for f in folds:
        tr = np.setdiff1d(idx, f)
        mt, _ = posterior(design(x[tr], M), y[tr], mm, SS)
        errs.append(rmse(y[f], design(x[f], M) @ mt))
    cvs.append(float(np.mean(errs)))
    mN, _ = posterior(Pm, y, mm, SS)
    tes.append(rmse(yte, design(xte, M) @ mN))
    print(f"{M:>4} {evs[-1]:>15.6f} {cvs[-1]:>16.6f} {tes[-1]:>12.6f}")
print(f"\nevidence picks M = {int(np.argmax(evs))}")
print(f"5-fold CV picks M = {int(np.argmin(cvs))}")
print(f"the 200-point test set picks M = {int(np.argmin(tes))}")

text

   M    log p(Y | X)   5-fold CV RMSE    test RMSE
   0     -115.019341         1.044644     0.893609
   1      -67.969272         0.843495     0.745023
   2      -46.997738         0.627236     0.686147
   3      -43.237138         1.127103     0.777397
   4      -24.465648         0.466187     0.322216
   5      -31.152704         2.498905     0.325011
   6      -35.307731         4.119212     0.453977
   7      -41.774266        12.570587     0.912259
   8      -48.800734        40.750308     5.740446
 
evidence picks M = 4
5-fold CV picks M = 4
the 200-point test set picks M = 4

Answer. All three agree on $M = 4$ .

But read the columns, not just the argmins. The evidence is smooth and gently curved. The CV column is violent — $0.466$ at $M = 4$ , then $2.499$ , $4.119$ , $12.571$ , $40.750$ . With ten points, five folds means training on eight, and a degree-8 model on eight points is interpolating; the held-out point is then predicted by a wildly oscillating curve.

So the two criteria agree here for different reasons. The evidence penalises capacity gently and smoothly; cross-validation punishes it savagely and noisily. Page 808 measured the same gentleness from the other side — every degree from 3 to 11 was “barely worth mentioning” against the best.

And the evidence is $\mathbf{45\times}$ cheaper here: one determinant per degree against $K_5$ refits.

Problem 4 — Choose where to measure, before measuring

Statement. Page 907 measured that $\mathbf{S}_N$ contains no $\mathbf{y}$ — the predictive variance is available before any target is observed. Use it: you may place ten input locations in $[-5,5]$ and must predict well at $x = -4.5$ , $0$ and $4.5$ . Which design?

Method. For each candidate set of locations, build $\mathbf{S}_N$ and evaluate the worst predictive half-width across the three targets. No targets are generated.

problem_4.py

targets = np.array([-4.5, 0.0, 4.5])
def worst_sd(xlocs, M=4, b2=0.4):
    Pl = design(xlocs, M)
    SN = np.linalg.inv(np.eye(M+1)/b2 + Pl.T @ Pl / SIG**2)
    Pt = design(targets, M)
    return float(np.sqrt(np.einsum("ij,jk,ik->i", Pt, SN, Pt)
                         + SIG**2).max())
 
plans = {
    "10 points clustered in [-1, 1]": np.linspace(-1, 1, 10),
    "10 points spread over [-5, 5]":  np.linspace(-5, 5, 10),
    "10 points at the two ends":      np.r_[np.linspace(-5, -4, 5),
                                            np.linspace(4, 5, 5)],
    "10 Chebyshev points on [-5, 5]": 5*np.cos(
        (2*np.arange(1, 11) - 1) * np.pi / 20),
}
print(f"{'experimental design':>34} {'worst 95% half-width':>22}")
for name, loc in plans.items():
    print(f"{name:>34} {1.959963984540054*worst_sd(loc):>22.6f}")

text

               experimental design   worst 95% half-width
    10 points clustered in [-1, 1]             286.948028
     10 points spread over [-5, 5]               0.473974
         10 points at the two ends               1.270762
    10 Chebyshev points on [-5, 5]               0.476172

Answer. A factor of $605$ between the best and worst design, decided entirely in advance.

design	worst half-width	reading
clustered in $[-1,1]$	$\mathbf{286.948028}$	a degree-4 fit from a narrow window extrapolates catastrophically
spread over $[-5,5]$	$\mathbf{0.473974}$	best
both ends only	$1.270762$	the middle is unconstrained
Chebyshev	$0.476172$	essentially tied with uniform

The clustered design is the lesson. Ten points is ten points, and the fit will look excellent on all of them — but $\phi^\top\mathbf{S}_N\phi$ at $x = \pm 4.5$ is enormous, because nothing constrains the high-order coefficients away from the cluster.

This is optimal experimental design in three lines, and it works only because $\mathbf{S}_N$ is a function of the inputs alone. Page 907 framed that as a limitation — the bars do not widen when the model fits badly. Here it is the corresponding feature.

Note also that Chebyshev, which is optimal for polynomial interpolation, ties with uniform here and does not beat it. The objective is different: minimising predictive variance at three specific targets, not minimising a worst-case interpolation error.

Problem 5 — How much wider is the honest interval?

Statement. Equation 9.6’s interval has half-width $z\sigma$ ; Equation 9.57’s has $z\sqrt{\phi^\top\mathbf{S}_N\phi + \sigma^2}$ . Quantify the ratio as a function of where you ask.

Method. The ratio is $\sqrt{1 + \phi^\top\mathbf{S}_N\phi/\sigma^2}$ . Evaluate it from the centre of the data out to well beyond it.

problem_5.py

Pm = design(x, 4)
mm, SS = np.zeros(5), 0.4 * np.eye(5)
mN, SN = posterior(Pm, y, mm, SS)
z = 1.959963984540054
print(f"{'x*':>7} {'plug-in half-width':>20} {'Eq 9.57 half-width':>20} "
      f"{'ratio':>10}")
for xv in (0.0, 2.0, 4.0, 5.0, 6.0, 8.0):
    ph = design([xv], 4)[0]
    full = np.sqrt(float(ph @ SN @ ph) + SIG**2)
    print(f"{xv:>7.1f} {z*SIG:>20.6f} {z*full:>20.6f} {full/SIG:>10.4f}")

text

     x*   plug-in half-width   Eq 9.57 half-width      ratio
    0.0             0.391993             0.446452     1.1389
    2.0             0.391993             0.460580     1.1750
    4.0             0.391993             0.453104     1.1559
    5.0             0.391993             0.611905     1.5610
    6.0             0.391993             1.674553     4.2719
    8.0             0.391993             7.754791    19.7830

Answer. $\sqrt{1 + \phi^\top\mathbf{S}_N\phi/\sigma^2}$ , which is

1.14\times \text{ at } x = 0, \qquad 1.16\times \text{ at } x = 4, \qquad \mathbf{19.78\times} \text{ at } x = 8

Inside the data the two intervals are nearly the same — $14\%$ apart at the centre — which is why Equation 9.6 survives so much routine use. The training inputs span roughly $[-4.2, 4.8]$ , and the ratio stays under $1.2$ across all of it.

Two units past the data it is $4.27\times$ ; four units past, $19.78\times$ . Page 907 measured the consequence: $0.1112$ coverage for the plug-in just outside the range, against a claimed $0.95$ .

The practical rule is worth stating: the plug-in interval is defensible for interpolation and indefensible for extrapolation, and the crossover is not gradual — it is quadratic in the feature magnitudes.

Problem 6 — Is the projection special to polynomials?

Statement. §9.4 develops the projection view with monomial features. Does any of it depend on that? Repeat every check with six Gaussian bumps instead.

Method. Build both feature matrices, form $P = \boldsymbol\Phi(\boldsymbol\Phi^\top\boldsymbol\Phi)^{-1}\boldsymbol\Phi^\top$ , and test idempotence, symmetry, trace, orthogonality and Pythagoras for each.

problem_6.py

def rbf(xv, centres, width=1.2):
    xv = np.asarray(xv, float)[:, None]
    return np.exp(-0.5 * ((xv - centres[None, :]) / width) ** 2)
 
centres = np.linspace(-4, 4, 6)
for name, A in (("degree-5 monomials", design(x, 5)),
                ("6 Gaussian bumps", rbf(x, centres))):
    Pmat = A @ np.linalg.solve(A.T @ A, A.T)
    th = np.linalg.lstsq(A, y, rcond=None)[0]
    r = y - A @ th
    print(f"\n{name}:")
    print(f"  max |P P - P|            : {np.abs(Pmat @ Pmat - Pmat).max():.3e}")
    print(f"  max |P - P^T|            : {np.abs(Pmat - Pmat.T).max():.3e}")
    print(f"  trace(P)                 : {np.trace(Pmat):.8f}  "
          f"(K = {A.shape[1]})")
    print(f"  max |A^T residual|       : {np.abs(A.T @ r).max():.3e}")
    print(f"  ||y||^2 - ||Py||^2 - ||r||^2 : "
          f"{abs(float(y@y) - float((Pmat@y)@(Pmat@y)) - float(r@r)):.3e}")

text

degree-5 monomials:
  max |P P - P|            : 1.810e-14
  max |P - P^T|            : 9.472e-15
  trace(P)                 : 6.00000000  (K = 6)
  max |A^T residual|       : 5.799e-11
  ||y||^2 - ||Py||^2 - ||r||^2 : 5.207e-14
 
6 Gaussian bumps:
  max |P P - P|            : 7.494e-16
  max |P - P^T|            : 2.637e-16
  trace(P)                 : 6.00000000  (K = 6)
  max |A^T residual|       : 1.104e-15
  ||y||^2 - ||Py||^2 - ||r||^2 : 3.886e-16

Answer. Nothing depends on polynomials. Both bases give a symmetric idempotent projection of rank $6$ with trace exactly $6.00000000$ , an orthogonal residual, and an exact Pythagoras split.

And the Gaussian bumps are better conditioned by four to five orders of magnitude: the orthogonality residual is $1.10\times10^{-15}$ against $5.80\times10^{-11}$ , and idempotence $7.49\times10^{-16}$ against $1.81\times10^{-14}$ .

That is page 909’s point arriving from a third direction. The subspace is what matters; the basis is a coordinate system; and some coordinate systems are numerically much better behaved than others — monomials being among the worst available, since $x^5$ and $x^4$ are nearly parallel over a bounded interval while Gaussian bumps at separated centres are nearly orthogonal.

Problem 7 — Fix Equation 9.22

Statement. Page 902 measured Equation 9.22’s bias and page 909 explained it: $\mathrm{tr}(\mathbf{I}-P) = N-K$ . Build the corrected estimator and verify it is unbiased at four sample sizes.

Method. Divide the residual sum of squares by $N-K$ instead of $N$ , and average over $40{,}000$ trials.

problem_7.py

TR = 40_000
print(f"{'N':>6} {'K':>4} {'E[s/N]':>12} {'E[s/(N-K)]':>13} "
      f"{'true sigma^2':>13}")
for n, M in ((10, 4), (20, 4), (50, 4), (200, 4)):
    Kk = M + 1
    rr = np.random.default_rng(55)
    xg = np.linspace(-5, 5, n)
    Pg = design(xg, M)
    base = Pg @ np.arange(1.0, Kk + 1.0) / Kk
    a = b = 0.0
    for _ in range(TR):
        yy = base + SIG * rr.standard_normal(n)
        rv = yy - Pg @ np.linalg.lstsq(Pg, yy, rcond=None)[0]
        s = float(rv @ rv)
        a += s / n                       # Equation 9.22
        b += s / (n - Kk)                # the correction
    print(f"{n:>6} {Kk:>4} {a/TR:>12.8f} {b/TR:>13.8f} {SIG**2:>13.8f}")

text

     N    K       E[s/N]    E[s/(N-K)]  true sigma^2
    10    5   0.02008692    0.04017384    0.04000000
    20    5   0.03009548    0.04012731    0.04000000
    50    5   0.03604815    0.04005351    0.04000000
   200    5   0.03897049    0.03996974    0.04000000

Answer. Dividing by $N-K$ removes the bias at every sample size, not asymptotically.

$N$	Eq 9.22	corrected	truth
$10$	$0.02008692$ ( $50.2\%$ )	$\mathbf{0.04017384}$	$0.04$
$20$	$0.03009548$ ( $75.2\%$ )	$\mathbf{0.04012731}$	$0.04$
$50$	$0.03604815$ ( $90.1\%$ )	$\mathbf{0.04005351}$	$0.04$
$200$	$0.03897049$ ( $97.4\%$ )	$\mathbf{0.03996974}$	$0.04$

The corrected column sits within Monte Carlo error of $0.04$ throughout. The uncorrected one converges only as $N$ grows, and at $N = 10$ with $K = 5$ it is half the truth.

The one-line fix matters because $\sigma^2$ propagates. Every interval in this chapter is built from it — Equation 9.6’s, Equation 9.38’s, Equation 9.57’s — and an interval built on half the correct variance is $\sqrt{2} \approx 1.41$ times too narrow before any other consideration.

Problem 8 — What does a prior do to a singular design?

Statement. Page 903 showed that rank-deficient $\boldsymbol\Phi$ leaves infinitely many tied estimators; page 904 showed the prior makes the estimate unique. Which one does it pick, and why that one? Test with two exactly duplicated columns.

Method. Duplicate a column of $\boldsymbol\Phi$ , try maximum likelihood, then compute the MAP estimate at four prior widths and inspect the two tied coefficients.

problem_8.py

Pd = design(x, 4)
Pd = np.column_stack([Pd, Pd[:, 2]])          # an exactly duplicated column
print(f"Phi has {Pd.shape[1]} columns and rank "
      f"{int(np.linalg.matrix_rank(Pd))}")
try:
    np.linalg.solve(Pd.T @ Pd, Pd.T @ y)
    print("maximum likelihood: solved (it should not have)")
except np.linalg.LinAlgError as e:
    print(f"maximum likelihood: LinAlgError -- {e}")
 
Kd = Pd.shape[1]
for b2v in (1e-2, 1.0, 1e2, 1e6):
    md, Sd = posterior(Pd, y, np.zeros(Kd), b2v * np.eye(Kd))
    print(f"  MAP with b^2 = {b2v:>7.0e}: ||theta|| = "
          f"{np.linalg.norm(md):>10.6f}, "
          f"theta[2] - theta[5] = {md[2]-md[5]:>12.3e}")

text

Phi has 6 columns and rank 5
maximum likelihood: LinAlgError -- Singular matrix
  MAP with b^2 =   1e-02: ||theta|| =   0.449601, theta[2] - theta[5] =    1.776e-15
  MAP with b^2 =   1e+00: ||theta|| =   0.957617, theta[2] - theta[5] =   -4.547e-13
  MAP with b^2 =   1e+02: ||theta|| =   0.969039, theta[2] - theta[5] =    2.910e-11
  MAP with b^2 =   1e+06: ||theta|| =   0.969154, theta[2] - theta[5] =    0.000e+00

Answer. It splits the tied weight exactly evenly, at every prior width.

Maximum likelihood raises LinAlgError: Singular matrix — $\boldsymbol\Phi^\top\boldsymbol\Phi$ is not invertible, exactly as §9.2.1’s rank condition warns. MAP always succeeds, and $\theta_2 - \theta_5$ is zero to machine precision in all four rows.

Why even? The two columns are identical, so the likelihood depends only on $\theta_2 + \theta_5$ — it is completely flat along the direction $(\ldots, +1, \ldots, -1)$ . The prior penalises $\theta_2^2 + \theta_5^2$ , and for a fixed sum that is minimised when the two are equal. The prior is not adding information about the data; it is choosing the minimum-norm point on a flat ridge, which is page 906’s observation in its sharpest form.

Note the norms: $0.4496$ , $0.9576$ , $0.9690$ , $0.9692$ . As $b^2 \to \infty$ the answer converges to the minimum-norm least-squares solution — the same thing np.linalg.lstsq returns silently. Page 903 flagged that as an unannounced convention; here it is visible as the limit of a prior you can state.

🧪 Try It Yourself

Exercise 1 – Which lambda?

Exercise 2 – A data stream

Exercise 3 – Design the experiment

Exercise 4 – A projection for any basis

Exercise 5 – Fix Equation 9.22

pch.quizTag Did the problems land?

Two different penalised objectives both reproduced Equation 9.31's MAP estimate. What differed?

pch.quizShowAnswer

B — Their lambdas, by a factor of N — because one normalises the data-fit term by N and the other does not — Lambda equal to 0.1 works for Equation 9.32 and 0.01 for Equation 8.12, on the same data with the same prior. Lambda is a property of the pair (objective, prior), not of the model — so quoting one without the other says nothing.
Processing 500 observations one at a time, how close did the result come to the batch computation?

pch.quizShowAnswer

B — 6.772e-15 — and after ONE observation the posterior sd was still 0.632, essentially the prior's — Precisions add, so the order and grouping cannot matter. And the first row is worth reading: one observation cannot constrain five parameters, and what keeps the answer finite is entirely the prior.
The evidence, 5-fold cross-validation and a 200-point test set all picked M = 4. What differed between them?

pch.quizShowAnswer

B — The shape: the evidence curve is smooth and gently curved, while the CV column runs 0.466, 2.499, 4.119, 12.571, 40.750 — With ten points, five folds means training on eight, and a degree-8 model on eight points is interpolating. The evidence penalises capacity gently and smoothly; cross-validation punishes it savagely and noisily. They agreed here for different reasons.
Ten input locations were chosen in advance to minimise the worst predictive interval at three targets. What was the spread between designs?

pch.quizShowAnswer

B — A factor of 605: 286.948028 for points clustered in [-1, 1] against 0.473974 for points spread over [-5, 5] — And no targets were measured to compute any of it, because S_N is a function of the inputs alone. Page 907 framed that property as a limitation — the bars do not widen when the model fits badly — and this is the corresponding feature.
With two exactly duplicated columns, maximum likelihood raised a singular-matrix error. What did MAP do?

pch.quizShowAnswer

B — It split the tied weight exactly evenly — the two coefficients differed by 0.000e+00 at every prior width — The likelihood depends only on the SUM of the two coefficients, so it is flat along their difference. The prior penalises the sum of squares, which for a fixed sum is minimised when they are equal. The prior is not adding information about the data; it is choosing the minimum-norm point on a flat ridge.
How much wider is Equation 9.57's interval than Equation 9.6's, and where?

pch.quizShowAnswer

B — 1.14 times at the centre of the data and 19.78 times four units outside it — The ratio is the square root of one plus phi-transpose S_N phi over sigma squared. Inside the data the two are within 20 percent, which is why the plug-in survives routine use. The practical rule: defensible for interpolation, indefensible for extrapolation, with a quadratic crossover.

Recall card

Lambda is a property of the pair (objective, prior), not of the model. Measured: 0.1 for Equation 9.32’s un-normalised loss and 0.01 for Equation 8.12’s, both reproducing the same MAP estimate on the same data.
Equation 9.33’s lambda matches the penalty terms and not the estimators — off by 0.2388 here. The missing factor is 2 sigma squared.
A 500-point stream can be absorbed one point at a time, matching the batch answer to 6.8e-15 in fixed memory, with no design matrix ever assembled.
After one observation with five parameters, almost nothing is learned: the largest posterior sd is 0.632 against a prior 0.632.
The evidence, 5-fold cross-validation and a held-out test set all chose degree 4 on the same ten points.
But the evidence is smooth and cross-validation is violent: CV RMSE runs 0.466, 2.499, 4.119, 12.571, 40.750 across degrees 4 to 8, because five folds on ten points means training a rich model on eight.
The predictive variance can be minimised before measuring anything. Ten points clustered in a narrow window give a worst 95 percent half-width of 286.95; the same ten spread out give 0.474 — a factor of 605.
That is optimal experimental design, and it works only because S_N contains no targets.
Equation 9.57’s interval is 1.14 times the plug-in’s at the centre of the data and 19.78 times four units outside it. The plug-in is defensible for interpolation and indefensible for extrapolation.
The projection view does not depend on polynomials. Gaussian bumps give the same symmetric idempotent rank-K projection with trace exactly K — and are four to five orders of magnitude better conditioned.
Dividing the residual sum of squares by N minus K removes the bias exactly, at every sample size: 0.04017, 0.04013, 0.04005, 0.03997 against a true 0.04.
A duplicated column makes maximum likelihood raise a singular-matrix error, while MAP splits the tied weight exactly evenly — a difference of 0.000e+00 at every prior width.
The prior does not add information about the data there. It chooses the minimum-norm point on a ridge the likelihood cannot see along, which is also what np.linalg.lstsq returns silently.

Next: the whole chapter on one page. Chapter 9 Formula Sheet

pch.coffeeTagline

pch.coffeeCta

pch.feedbackHeading

pch.feedbackSubheading