Chapter 9 Formula Sheet

One page for the whole chapter. Equation numbers are the book’s; the measured columns are this module’s.

The chapter in one table

Three estimators, one linear system, three settings of the prior.

	maximum likelihood	MAP	Bayesian
section	§9.2.1	§9.2.3	§9.3
matrix inverted	$\boldsymbol\Phi^\top\boldsymbol\Phi$	$\boldsymbol\Phi^\top\boldsymbol\Phi + \tfrac{\sigma^2}{b^2}\mathbf{I}$	$\mathbf{S}_0^{-1} + \sigma^{-2}\boldsymbol\Phi^\top\boldsymbol\Phi$
equation	9.19	9.31	9.43
the answer	a point	a point	a distribution
exists when $K > N$	no	yes	yes
predictive variance	$\sigma^2$	$\sigma^2$	$\phi^\top\mathbf{S}_N\phi + \sigma^2$
measured coverage	$0.7419$ / $0.1112$	same	$\mathbf{0.9487}$ / $\mathbf{0.9499}$
the prediction curve	its own	—	identical to MAP, to $8.4\times10^{-14}$

§9.1 Problem formulation

what	equation
the likelihood	$p(y \mid \mathbf{x}) = \mathcal{N}(y \mid f(\mathbf{x}), \sigma^2)$ — 9.1
equivalently	$y = f(\mathbf{x}) + \epsilon$, $\epsilon \sim \mathcal{N}(0,\sigma^2)$ i.i.d. — 9.2
the linear case	$p(y \mid \mathbf{x},\boldsymbol\theta) = \mathcal{N}(y \mid \mathbf{x}^\top\boldsymbol\theta, \sigma^2)$ — 9.3, 9.4
with features	$y = \phi^\top(\mathbf{x})\boldsymbol\theta + \epsilon = \sum_{k=0}^{K-1}\theta_k\phi_k(\mathbf{x}) + \epsilon$ — 9.13
monomial features	$\phi(x) = [1, x, x^2, \ldots, x^{K-1}]^\top$ — 9.14
the feature matrix	$\boldsymbol\Phi \in \mathbb{R}^{N\times K}$, $\Phi_{ij} = \phi_j(x_i)$ — 9.16

Equation 9.4 has no intercept

“The class of functions described by (9.4) are straight lines that pass through the origin.” Measured on data with a true intercept of $4.0$: RMSE $4.091288$ against $0.514303$, and the slope comes out at $1.193712$ against a true $1.3$. A missing feature corrupts the features you kept.

“Linear regression” constrains theta, not x

Measured on a degree-9 model: superposition holds in $\boldsymbol\theta$ to $4.366\times10^{-11}$ and fails in $x$ by $2.341\times10^{5}$ — sixteen orders of magnitude apart. That gap licenses Equation 9.19 being Equation 9.12c with $\mathbf{X}$ replaced by $\boldsymbol\Phi$.

The likelihood is a density in y, not in theta

integral	value
$\int p(y \mid x, \theta)\,dy$	$1.00000000$
$\int p(y \mid x, \theta)\,d\theta$	exactly $1/\lvert x\rvert$ — measured $0.50000000$ at $x=2$

It reaches $1$ only when $\lvert x\rvert = 1$ and diverges as $x \to 0$. The missing normaliser is what §9.3.3 supplies and §9.3.5 computes.

§9.2 Parameter estimation

what	equation
the likelihood factorizes	$p(\mathcal{Y}\mid\mathcal{X},\boldsymbol\theta) = \prod_{n}\mathcal{N}(y_n \mid \mathbf{x}_n^\top\boldsymbol\theta, \sigma^2)$ — 9.5
the plug-in predictive	$p(y_* \mid \mathbf{x}_*, \boldsymbol\theta^*) = \mathcal{N}(y_* \mid \mathbf{x}_*^\top\boldsymbol\theta^*, \sigma^2)$ — 9.6
the error function	$\mathcal{L}(\boldsymbol\theta) = \frac{1}{2\sigma^2}\lVert\mathbf{y}-\mathbf{X}\boldsymbol\theta\rVert^2$ — 9.10
its gradient	$\frac{1}{\sigma^2}(-\mathbf{y}^\top\mathbf{X} + \boldsymbol\theta^\top\mathbf{X}^\top\mathbf{X})$ — 9.11c
the normal equations	$\boldsymbol\theta_{\text{ML}} = (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y}$ — 9.12c
with features	$\boldsymbol\theta_{\text{ML}} = (\boldsymbol\Phi^\top\boldsymbol\Phi)^{-1}\boldsymbol\Phi^\top\mathbf{y}$ — 9.19
the noise variance	$\sigma^2_{\text{ML}} = \frac{1}{N}\sum_n (y_n - \phi^\top(\mathbf{x}_n)\boldsymbol\theta)^2$ — 9.22
the RMSE	$\sqrt{\frac{1}{N}\lVert\mathbf{y}-\boldsymbol\Phi\boldsymbol\theta\rVert^2}$ — 9.23

Solvability: $\boldsymbol\Phi^\top\boldsymbol\Phi$ is invertible iff $\mathrm{rk}(\boldsymbol\Phi) = K$, which requires $K \leqslant N$ and no redundant features.

Never form the Gram matrix

Measured, $\kappa(\boldsymbol\Phi^\top\boldsymbol\Phi) = \kappa(\boldsymbol\Phi)^2$ to six decimal places at every degree. Against a known $\boldsymbol\theta$:

degree	$\kappa(\boldsymbol\Phi)$	normal equations	lstsq	ratio
$8$	$3.162\times10^{5}$	$3.207\times10^{-12}$	$5.517\times10^{-13}$	$5.8\times$
$12$	$3.262\times10^{8}$	$3.405\times10^{-8}$	$4.665\times10^{-9}$	$7.3\times$
$16$	$3.922\times10^{11}$	$3.313\times10^{-5}$	$2.861\times10^{-8}$	$\mathbf{1157.7\times}$
$20$	$4.988\times10^{14}$	$3.868\times10^{-2}$	$4.757\times10^{-1}$	both failed

Equation 9.12c is a derivation; np.linalg.lstsq is the implementation.

Equation 9.22 is biased by exactly (N-K)/N

$N$	$K$	$\mathbb{E}[\hat\sigma^2_{\text{ML}}]/\sigma^2$	$(N-K)/N$
$10$	$5$	$\mathbf{0.500606}$	$0.500000$
$20$	$5$	$0.750030$	$0.750000$
$200$	$5$	$0.974591$	$0.975000$
$1000$	$5$	$0.994725$	$0.995000$

Divide by $N-K$. Verified unbiased: $0.04017$, $0.04013$, $0.04005$, $0.03997$ against a true $0.04$. §9.4 explains why — $\mathrm{tr}(\mathbf{I}-P) = N-K$.

§9.2.2 Overfitting

Figure 9.6, reproduced

$M$	training RMSE	test RMSE	$\lVert\boldsymbol\theta\rVert$
$1$	$0.743590$	$0.745097$	$0.2150$
$\mathbf{4}$	$0.270626$	$\mathbf{0.324947}$	$0.9904$
$6$	$0.207586$	$0.473424$	$1.1985$
$9$	$\mathbf{6.506\times10^{-11}}$	$\mathbf{100.850623}$	$4.2030$

The training error never increases — nine steps, zero increases. It is a theorem from set inclusion, so it carries no information. At $M = N-1$ the fit is exact and generalization is destroyed.

And Figure 9.6 is one draw. Over $2000$ repeats, $M = 4$ wins only $46.1\%$ of the time; $M = 6$ wins $20.8\%$; $M = 1$ wins $11.3\%$.

Search bound: $0 \leqslant M \leqslant N-1$. Past it $\boldsymbol\Phi$ has a null space — measured dimension $1$ at $M = 10$, $N = 10$ — and estimators with norms from $3.44$ to $5000.00$ fit equally well. lstsq returns the minimum-norm one by convention.

§9.2.3–9.2.4 MAP and regularization

what	equation
the posterior	$p(\boldsymbol\theta \mid \mathcal{X},\mathcal{Y}) \propto p(\mathcal{Y}\mid\mathcal{X},\boldsymbol\theta)p(\boldsymbol\theta)$ — 9.24
log-posterior	$= \log p(\mathcal{Y}\mid\mathcal{X},\boldsymbol\theta) + \log p(\boldsymbol\theta) + \text{const}$ — 9.25
the MAP estimate	$\boldsymbol\theta_{\text{MAP}} = \left(\boldsymbol\Phi^\top\boldsymbol\Phi + \tfrac{\sigma^2}{b^2}\mathbf{I}\right)^{-1}\boldsymbol\Phi^\top\mathbf{y}$ — 9.31
regularized least squares	$\lVert\mathbf{y}-\boldsymbol\Phi\boldsymbol\theta\rVert^2 + \lambda\lVert\boldsymbol\theta\rVert_2^2$ — 9.32
the negative log-prior	$-\log p(\boldsymbol\theta) = \frac{1}{2b^2}\lVert\boldsymbol\theta\rVert_2^2 + \text{const}$ — 9.33
its solution	$\boldsymbol\theta_{\text{RLS}} = (\boldsymbol\Phi^\top\boldsymbol\Phi + \lambda\mathbf{I})^{-1}\boldsymbol\Phi^\top\mathbf{y}$ — 9.34

Three lambdas for one idea

objective	$\lambda$	reproduces Eq 9.31?
Eq 9.32, $\lVert\cdot\rVert^2 + \lambda\lVert\boldsymbol\theta\rVert^2$	$\boldsymbol\sigma^2/b^2$	yes, $0.00\times10^{0}$
Eq 9.33’s stated value	$1/(2b^2)$	no — off by $1.87\times10^{-1}$
Eq 8.12, $\frac1N\lVert\cdot\rVert^2 + \lambda\lVert\boldsymbol\theta\rVert^2$	$\sigma^2/(N\tau^2)$	yes, $7.8\times10^{-15}$

Equation 9.33’s statement matches the two penalty terms; Equation 9.34’s matches the two estimators, and the difference is a factor of $2\sigma^2$. A $\lambda$ is meaningless without its objective.

What the extra term does

It adds $\sigma^2/b^2$ to every eigenvalue. Measured at $M = 12$, $N = 10$: a smallest eigenvalue of exactly $0$ becomes $0.04$, and an infinite condition number becomes finite. At $M = 4$ the condition number barely moves ($1.817\times10^{5} \to 1.794\times10^{5}$) — it is not a preconditioner.

Example 9.6 measured: at $M = 9$ the prior cuts $\lVert\boldsymbol\theta\rVert$ from $4.2030$ to $1.2062$ and test RMSE from $100.8506$ to $29.9717$. But best MAP is $0.323740$ against best MLE’s $0.324947$ — $0.4\%$ — and at $M = 7$ MAP is worse by $1.1652\times$.

The $p$-norm remark. Smaller $p$ gives sparser solutions; $p=1$ is LASSO. Measured on 30 features with 3 real ones: ridge produced zero exact zeros at $\lambda = 2, 8, 20$; LASSO produced $17$ then $27$ (the truth has $27$). A quadratic penalty has zero gradient at the origin; $\ell_1$ has a kink.

§9.3 Bayesian linear regression

what	equation
the model	$p(\boldsymbol\theta) = \mathcal{N}(\mathbf{m}_0,\mathbf{S}_0)$, $\ p(y\mid\mathbf{x},\boldsymbol\theta) = \mathcal{N}(\phi^\top(\mathbf{x})\boldsymbol\theta,\sigma^2)$ — 9.35
the joint	$p(y,\boldsymbol\theta\mid\mathbf{x}) = p(y\mid\mathbf{x},\boldsymbol\theta)p(\boldsymbol\theta)$ — 9.36
prior predictive	$\mathcal{N}\!\left(\phi^\top\mathbf{m}_0,\ \phi^\top\mathbf{S}_0\phi + \sigma^2\right)$ — 9.38
variance decomposition	$\mathbb{V}[y_*] = \mathbb{V}_{\boldsymbol\theta}[\phi^\top\boldsymbol\theta] + \mathbb{V}_\epsilon[\epsilon]$ — 9.39
noise-free version	$\mathcal{N}\!\left(\phi^\top\mathbf{m}_0,\ \phi^\top\mathbf{S}_0\phi\right)$ — 9.40
the posterior	$p(\boldsymbol\theta\mid\mathcal{X},\mathcal{Y}) = \dfrac{p(\mathcal{Y}\mid\mathcal{X},\boldsymbol\theta)p(\boldsymbol\theta)}{p(\mathcal{Y}\mid\mathcal{X})}$ — 9.41
the marginal likelihood	$p(\mathcal{Y}\mid\mathcal{X}) = \int p(\mathcal{Y}\mid\mathcal{X},\boldsymbol\theta)p(\boldsymbol\theta)\,d\boldsymbol\theta$ — 9.42

Theorem 9.1

$$\boxed{\ \mathbf{S}_N = \left(\mathbf{S}_0^{-1} + \sigma^{-2}\boldsymbol\Phi^\top\boldsymbol\Phi\right)^{-1}, \qquad \mathbf{m}_N = \mathbf{S}_N\left(\mathbf{S}_0^{-1}\mathbf{m}_0 + \sigma^{-2}\boldsymbol\Phi^\top\mathbf{y}\right)\ }$$

property	measured
verified as a posterior (log gap constant in $\boldsymbol\theta$)	spread $1.805\times10^{-12}$
that constant	$-\log p(\mathcal{Y}\mid\mathcal{X}) = \mathbf{30.453407}$
grid cross-check, $1400\times1400$, 2-D	mean, sd, correlation to $10^{-8}$
precisions add	$\max\lvert\mathbf{S}_N\mathbf{A}-\mathbf{I}\rvert = 6.9\times10^{-11}$
sequential updating	batch vs one-at-a-time: $4.1\times10^{-12}$; over 500 points, $6.8\times10^{-15}$
$\mathbf{m}_N$ vs $\boldsymbol\theta_{\text{MAP}}$	$1.19\times10^{-13}$
$\mathbf{m}_N$ vs $\boldsymbol\theta_{\text{ML}}$ at $b^2 = 10^8$	$1.43\times10^{-10}$
contraction	sd $\times\sqrt{N} \to 0.37$ past $N = 50$
posterior correlations (prior’s are $\mathbf{I}$)	up to $\mathbf{0.969560}$

§9.3.4 Posterior predictions

$$p(y_*\mid\mathcal{X},\mathcal{Y},\mathbf{x}_*) = \mathcal{N}\!\left(\phi^\top(\mathbf{x}_*)\mathbf{m}_N,\ \ \phi^\top(\mathbf{x}_*)\mathbf{S}_N\phi(\mathbf{x}_*) + \sigma^2\right) \qquad \textbf{(9.57)}$$

with $\mathbb{E}[y_*] = \phi^\top\mathbf{m}_N = \phi^\top\boldsymbol\theta_{\text{MAP}}$ — measured to $8.438\times10^{-14}$ over 400 inputs, while the MLE’s curve differs by $0.202578$.

S_N contains no targets

change to $\mathbf{y}$	$\max\lvert\mathbf{S}_N - \mathbf{S}_N^{\text{new}}\rvert$
shuffled	$\mathbf{0.000\times10^{0}}$
replaced with nonsense	$\mathbf{0.000\times10^{0}}$

while $\mathbf{m}_N$ moves by $0.734789$. Feature: error bars are computable during experimental design — measured, a clustered design gives a worst half-width of $286.948028$ against $0.473974$ for a spread one, a factor of $605$. Limitation: the shading does not widen when the model fits badly.

Calibration, 3000 trials

region	Eq 9.57	Eq 9.6 plug-in
inside $[-5,5]$	$\mathbf{0.9487}$	$0.7419$
just outside	$\mathbf{0.9499}$	$\mathbf{0.1112}$

Both share a mean. The ratio of half-widths is $\sqrt{1 + \phi^\top\mathbf{S}_N\phi/\sigma^2}$: measured $1.14\times$ at $x=0$ and $19.78\times$ at $x=8$.

§9.3.5 The marginal likelihood

$$\mathbb{E}[\mathcal{Y}\mid\mathcal{X}] = \mathbf{X}\mathbf{m}_0 \quad \textbf{(9.62)}, \qquad \mathrm{Cov}[\mathcal{Y}\mid\mathcal{X}] = \mathbf{X}\mathbf{S}_0\mathbf{X}^\top + \sigma^2\mathbf{I} \quad \textbf{(9.63)}$$ $$\boxed{\ p(\mathcal{Y}\mid\mathcal{X}) = \mathcal{N}\!\left(\mathbf{y}\ \middle|\ \mathbf{X}\mathbf{m}_0,\ \mathbf{X}\mathbf{S}_0\mathbf{X}^\top + \sigma^2\mathbf{I}\right)\ } \qquad \textbf{(9.64)}$$

Use the $K\times K$ form. Woodbury gives $\det(\mathbf{X}\mathbf{S}_0\mathbf{X}^\top + \sigma^2\mathbf{I}) = \det(\mathbf{S}_N^{-1})\det(\mathbf{S}_0)\sigma^{2N}$: agreeing to $8.9\times10^{-6}$ at $N=5000$ and $4341\times$ faster at $N=4000$.

It chooses the degree

$M$	$\log p(\mathcal{Y}\mid\mathcal{X})$	max log lik	test RMSE
$2$	$-46.492910$	$-36.6089$	$0.6848$
$\mathbf{4}$	$\mathbf{-23.985921}$	$-2.2498$	$\mathbf{0.3210}$
$6$	$-34.643032$	$1.5185$	$0.4440$
$9$	$-54.242980$	$\mathbf{6.9050}$	$16.7174$

The maximum log likelihood rises from $-112.66$ to $+6.91$ and never turns. The evidence peaks at $M=4$ — the same degree a 200-point test set picks, and the same one 5-fold CV picks, using none of it.

And it is hostage to the prior

Widening $b^2$ from $0.01$ to $10^8$ on identical data: degree 9 loses $\mathbf{91.07}$ nats, degree 1 loses $20.60$. The ordering of degrees 5 and 3 flips from $+10.21$ to $-6.42$. The winner held at $M=4$ here; page 808 measured a case where it did not.

§9.4 Maximum likelihood as orthogonal projection

what	equation
the scalar estimator	$\theta_{\text{ML}} = \dfrac{\mathbf{X}^\top\mathbf{y}}{\mathbf{X}^\top\mathbf{X}}$ — 9.66
the projection	$\mathbf{X}\theta_{\text{ML}} = \dfrac{\mathbf{X}\mathbf{X}^\top}{\mathbf{X}^\top\mathbf{X}}\mathbf{y}$ — 9.67
the general case	$P = \boldsymbol\Phi(\boldsymbol\Phi^\top\boldsymbol\Phi)^{-1}\boldsymbol\Phi^\top$ — 9.69, 9.70
orthonormal features	$\boldsymbol\Phi^\top\boldsymbol\Phi = \mathbf{I}$, so $P = \boldsymbol\Phi\boldsymbol\Phi^\top$

property	measured
idempotent, $PP = P$	$5.551\times10^{-17}$
symmetric, $P = P^\top$	$0.000\times10^{0}$
$\mathrm{tr}(P) = K$	exact to 8 decimals, every degree
residual orthogonal, $\boldsymbol\Phi^\top\mathbf{r} = \mathbf{0}$	$9.551\times10^{-15}$
Pythagoras, $\lVert\mathbf{y}\rVert^2 = \lVert P\mathbf{y}\rVert^2 + \lVert\mathbf{r}\rVert^2$	$8.53\times10^{-14}$

Why Equation 9.22 is biased, in one line

$$\mathbb{E}\!\left[\lVert(\mathbf{I}-P)\mathbf{y}\rVert^2\right] = \sigma^2\,\mathrm{tr}(\mathbf{I}-P) = \sigma^2(N-K)$$

Measured $0.200222$ against $\sigma^2(N-K) = 0.200000$, where Equation 9.22 assumes $\sigma^2 N = 0.400000$. The fit has already spent $K$ of the $N$ dimensions.

The basis is only a coordinate system

Monomials and an orthonormal basis for the same span: projection matrices agree to $2.384\times10^{-13}$, fitted values to $4.802\times10^{-13}$ — with $\kappa = 2.195\times10^{26}$ and $\mathbf{1.000000}$, and parameter norms of $1.198468$ and $3.096718$.

A ridge penalty acts on coordinates, so changing the basis changes what it penalises while leaving the unregularised fit untouched. Nothing here is special to polynomials: Gaussian bumps give trace $6$ and are $4$–$5$ orders of magnitude better conditioned.

§9.5 Further reading

pointer	what it says
generalized linear models	$x_{k+1} = \sigma_k(\mathbf{A}_kx_k + \mathbf{b}_k)$ — 9.72. A $K$-layer deep network is a composition of these.
neural networks	far more expressive, but maximum likelihood is non-convex and Bayesian marginalization is intractable
Gaussian processes	place a distribution directly on functions, “without the detour via the parameters”, via the kernel trick
non-Gaussian priors	for $N \ll D$ the problem is underdetermined; a Laplace prior enforces sparsity — equivalent to $\ell_1$, i.e. LASSO (Tibshirani, 1996)

The Gaussian process is already here

Page 905 measured that $p(\boldsymbol\theta) = \mathcal{N}(\mathbf{0},\mathbf{S}_0)$ induces the covariance function $k(x_a,x_b) = \phi(x_a)^\top\mathbf{S}_0\phi(x_b)$ — matched to sampling within $0.5\%$, including a negative value of $-204.75$ between $x=-2$ and $x=+2$. That function is the kernel §9.5 points to.

The constants this module measured

quantity	value	page
likelihood’s integral over $\theta$	exactly $1/\lvert x\rvert$	901
superposition, $\boldsymbol\theta$ vs $x$	$4.4\times10^{-11}$ vs $2.3\times10^{5}$	901
product of $N$ densities underflows at	$N = 1000$, exactly $0.0$	901
$\kappa(\boldsymbol\Phi^\top\boldsymbol\Phi)/\kappa(\boldsymbol\Phi)^2$	$1.000000$	902
normal equations vs lstsq at $M=16$	$1157.7\times$ worse	902
Eq 9.22 bias	exactly $(N-K)/N$	902
test RMSE at $M = N-1 = 9$	$100.850623$	903
$M=4$ wins over 2000 draws	$46.1\%$	903
$\lambda$ that makes RLS $=$ MAP	$\sigma^2/b^2$, to $0.00\times10^{0}$	904
ridge exact zeros, any $\lambda$	$\mathbf{0}$ of 30	904
95% prior band at $x=5$, degree 5	$3125.5935$	905
prior band fits the book’s axes at	$48$ of $200$ inputs	905
$\mathrm{Cov}[f(-2), f(2)]$	$-204.75$	905
Theorem 9.1’s constant offset	$30.453407$	906
sequential vs batch, 500 points	$6.772\times10^{-15}$	906, 910
posterior correlation, prior $=\mathbf{I}$	$0.969560$	906
Eq 9.57 vs Eq 9.6 coverage	$0.9487$/$0.9499$ vs $0.7419$/$0.1112$	907
$\mathbf{S}_N$ under shuffled targets	$0.000\times10^{0}$	907
$K\times K$ evidence speedup at $N=4000$	$4341\times$	908
evidence peak	$M = 4$, matching CV and the test set	908, 910
$\mathrm{tr}(P)$	$= K$, exact	909
$\mathbb{E}\lVert(\mathbf{I}-P)\mathbf{y}\rVert^2$	$0.200222$ vs $\sigma^2(N-K) = 0.2$	909
$\kappa$: monomial vs orthonormal, $M=18$	$2.195\times10^{26}$ vs $1.000000$	909
experimental design, clustered vs spread	$286.948028$ vs $0.473974$	910

The chapter in twelve lines

1. Model the noise, not the loss. Everything follows from Equation 9.1.
2. “Linear” constrains $\boldsymbol\theta$, not $x$ — to $4.4\times10^{-11}$ against $2.3\times10^{5}$.
3. Equation 9.4 has no intercept, and a missing feature corrupts the ones you kept.
4. The objective is quadratic, so one solve suffices and Chapter 7 is unnecessary — until you leave the class.
5. Write Equation 9.12c; call np.linalg.lstsq. The Gram matrix squares the condition number.
6. Equation 9.22 is biased by $(N-K)/N$, because $\mathrm{tr}(\mathbf{I}-P) = N-K$. Divide by $N-K$.
7. The training error cannot rise, so it carries no information about generalization.
8. A penalty is a prior — with $\lambda = \sigma^2/b^2$ for Equation 9.32, and never without naming the objective.
9. Precisions add, so the posterior update runs online in fixed memory, exactly.
10. MAP and Bayesian give the same curve, to $8.4\times10^{-14}$. The whole return is $\phi^\top\mathbf{S}_N\phi$.
11. That term is calibrated — $0.9487$ against $0.1112$ — and it never saw the targets, which makes it both a design tool and a blind spot.
12. A projection is a property of the subspace. The basis is a coordinate system, and some are $10^{26}$ times better than others.

Back to the start: Linear Regression Overview