One page for the whole chapter. Equation numbers are the book’s; the measured columns are this module’s.
Chapter 8’s own organising idea — §8.2, §8.3 and §8.4 are three answers to one question.
§8.2 empirical risk §8.3 maximum likelihood / MAP §8.4 Bayesian inference minimise / maximise R emp R_{\text{emp}} R emp − log p ( y ∣ θ ) -\log p(\mathbf{y} \mid \boldsymbol\theta) − log p ( y ∣ θ ) nothing — integrate needs a prior no MLE no, MAP yes yes needs p ( X ) p(\mathcal{X}) p ( X ) no no, it cancels yes , it is an integralcomputational problem optimization optimization integration the answer a point a point a distribution far-extrapolation coverage — 0.1796 0.1796 0.1796 0.9498 \mathbf{0.9498} 0.9498
term meaning data as vectors numerical, with a fixed dimension and an agreed ordering model as a function a predictor f : R D → R f: \mathbb{R}^D \to \mathbb{R} f : R D → R model as a distribution a probabilistic description of the process that made the data learning finding parameters that generalise, not that memorise
The three phases of learning: prediction / inference , training / parameter estimation , and
hyperparameter tuning / model selection .
what equation notes affine predictor f ( x n , θ ) = θ ⊤ x n f(\mathbf{x}_n, \boldsymbol\theta) = \boldsymbol\theta^\top\mathbf{x}_n f ( x n , θ ) = θ ⊤ x n Eq 8.2, with the bias folded in the unit-feature trick x n ← [ 1 , x n , 1 , … ] ⊤ \mathbf{x}_n \leftarrow [1, x_{n,1}, \ldots]^\top x n ← [ 1 , x n , 1 , … ] ⊤ Eq 8.4; makes affine look linear empirical risk R emp ( f , X , y ) = 1 N ∑ n = 1 N ℓ ( y n , y ^ n ) R_{\text{emp}}(f, \mathbf{X}, \mathbf{y}) = \frac{1}{N}\sum_{n=1}^{N}\ell(y_n, \hat{y}_n) R emp ( f , X , y ) = N 1 ∑ n = 1 N ℓ ( y n , y ^ n ) Eq 8.7 squared loss ℓ ( y n , y ^ n ) = ( y n − y ^ n ) 2 \ell(y_n, \hat{y}_n) = (y_n - \hat{y}_n)^2 ℓ ( y n , y ^ n ) = ( y n − y ^ n ) 2 Eq 8.6 least-squares risk 1 N ∥ y − X θ ∥ 2 \frac{1}{N}\lVert\mathbf{y} - \mathbf{X}\boldsymbol\theta\rVert^2 N 1 ∥ y − X θ ∥ 2 Eq 8.8 normal equations θ ∗ = ( X ⊤ X ) − 1 X ⊤ y \boldsymbol\theta^* = (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y} θ ∗ = ( X ⊤ X ) − 1 X ⊤ y Eq 8.9 expected risk R true ( f ) = E ( x , y ) [ ℓ ( y , f ( x ) ) ] R_{\text{true}}(f) = \mathbb{E}_{(\mathbf{x},y)}[\ell(y, f(\mathbf{x}))] R true ( f ) = E ( x , y ) [ ℓ ( y , f ( x ))] Eq 8.10 — what you actually wantthe gap R emp R_{\text{emp}} R emp R true R_{\text{true}} R true the whole chapter lives here
what equation notes regularized risk min θ 1 N ∥ y − X θ ∥ 2 + λ ∥ θ ∥ 2 \min_{\boldsymbol\theta}\ \frac{1}{N}\lVert\mathbf{y} - \mathbf{X}\boldsymbol\theta\rVert^2 + \lambda\lVert\boldsymbol\theta\rVert^2 min θ N 1 ∥ y − X θ ∥ 2 + λ ∥ θ ∥ 2 Eq 8.12 ridge solution θ = ( X ⊤ X N + λ I ) − 1 X ⊤ y N \boldsymbol\theta = \left(\frac{\mathbf{X}^\top\mathbf{X}}{N} + \lambda\mathbf{I}\right)^{-1}\frac{\mathbf{X}^\top\mathbf{y}}{N} θ = ( N X ⊤ X + λ I ) − 1 N X ⊤ y always invertible for λ > 0 \lambda > 0 λ > 0 ℓ 1 \ell_1 ℓ 1 λ ∥ θ ∥ 1 \lambda\lVert\boldsymbol\theta\rVert_1 λ ∥ θ ∥ 1 lasso sets coefficients to exactly zero the reading λ \lambda λ ∥ θ ∥ \lVert\boldsymbol\theta\rVert ∥ θ ∥ and equals σ 2 / ( N τ 2 ) \sigma^2/(N\tau^2) σ 2 / ( N τ 2 )
E V [ R ( V ∣ M ) ] ≈ 1 K ∑ k = 1 K R ( V ( k ) ∣ M ) (Eq 8.39) \mathbb{E}_{\mathcal{V}}[R(\mathcal{V} \mid M)] \approx \frac{1}{K}\sum_{k=1}^{K} R(\mathcal{V}^{(k)} \mid M) \qquad \text{(Eq 8.39)} E V [ R ( V ∣ M )] ≈ K 1 k = 1 ∑ K R ( V ( k ) ∣ M ) (Eq 8.39) with standard error σ / K \sigma/\sqrt{K} σ / K σ \sigma σ
K K K mean estimate spread bias vs truth 2 2 2 0.238485 0.238485 0.238485 0.146749 0.146749 0.146749 + 0.045981 +0.045981 + 0.045981 5 5 5 0.193586 0.193586 0.193586 0.055525 0.055525 0.055525 + 0.001082 +0.001082 + 0.001082 10 10 10 0.188288 0.188288 0.188288 0.048385 0.048385 0.048385 − 0.004215 -0.004215 − 0.004215 40 40 40 0.186229 0.186229 0.186229 0.045937 0.045937 0.045937 − 0.006274 -0.006274 − 0.006274
Small K K K overstates the risk. K = 5 K = 5 K = 5 10 10 10
what equation notes likelihood p ( y n ∣ x n , θ ) p(y_n \mid \mathbf{x}_n, \boldsymbol\theta) p ( y n ∣ x n , θ ) a function of θ \boldsymbol\theta θ negative log-likelihood L ( θ ) = − ∑ n = 1 N log p ( y n ∣ x n , θ ) \mathcal{L}(\boldsymbol\theta) = -\sum_{n=1}^{N}\log p(y_n \mid \mathbf{x}_n, \boldsymbol\theta) L ( θ ) = − ∑ n = 1 N log p ( y n ∣ x n , θ ) Eq 8.14 Gaussian noise p ( y n ∣ x n , θ ) = N ( y n ∣ x n ⊤ θ , σ 2 ) p(y_n \mid \mathbf{x}_n, \boldsymbol\theta) = \mathcal{N}(y_n \mid \mathbf{x}_n^\top\boldsymbol\theta, \sigma^2) p ( y n ∣ x n , θ ) = N ( y n ∣ x n ⊤ θ , σ 2 ) Eq 8.17 worked out L = 1 2 σ 2 ∑ n ( y n − x n ⊤ θ ) 2 + const \mathcal{L} = \frac{1}{2\sigma^2}\sum_n (y_n - \mathbf{x}_n^\top\boldsymbol\theta)^2 + \text{const} L = 2 σ 2 1 ∑ n ( y n − x n ⊤ θ ) 2 + const Eq 8.18d
L ( θ ) = N 2 σ 2 R emp ( θ ) + const \mathcal{L}(\boldsymbol\theta) = \dfrac{N}{2\sigma^2}R_{\text{emp}}(\boldsymbol\theta) + \text{const} L ( θ ) = 2 σ 2 N R emp ( θ ) + const affine function of the empirical risk, so identical minimisers. Measured agreement: 1.8 × 10 − 15 1.8\times10^{-15} 1.8 × 1 0 − 15
Least squares is a consequence of Gaussian noise. Laplace noise gives the median instead:
measured 3.420000 3.420000 3.420000 1.400000 1.400000 1.400000
Asymptotics (§8.3.2’s remark). The MLE is consistent, asymptotically normal, and its error variance
decays as 1 / N 1/N 1/ N “the size of the samples necessary to achieve these properties can be quite
large” , and “especially in the ‘small’ data regime, maximum likelihood estimation can lead to
overfitting.”
what equation notes Bayes for parameters p ( θ ∣ x ) = p ( x ∣ θ ) p ( θ ) p ( x ) p(\boldsymbol\theta \mid \mathbf{x}) = \dfrac{p(\mathbf{x} \mid \boldsymbol\theta)p(\boldsymbol\theta)}{p(\mathbf{x})} p ( θ ∣ x ) = p ( x ) p ( x ∣ θ ) p ( θ ) Eq 8.19 the proportionality p ( θ ∣ x ) ∝ p ( x ∣ θ ) p ( θ ) p(\boldsymbol\theta \mid \mathbf{x}) \propto p(\mathbf{x} \mid \boldsymbol\theta)p(\boldsymbol\theta) p ( θ ∣ x ) ∝ p ( x ∣ θ ) p ( θ ) Eq 8.20 — p ( x ) p(\mathbf{x}) p ( x ) θ \boldsymbol\theta θ MAP objective minimise − log p ( x ∣ θ ) − log p ( θ ) -\log p(\mathbf{x} \mid \boldsymbol\theta) - \log p(\boldsymbol\theta) − log p ( x ∣ θ ) − log p ( θ ) one extra term the correspondence λ = σ 2 N τ 2 \boxed{\lambda = \dfrac{\sigma^2}{N\tau^2}} λ = N τ 2 σ 2 p ( θ ) = N ( 0 , τ 2 I ) p(\boldsymbol\theta) = \mathcal{N}(\mathbf{0}, \tau^2\mathbf{I}) p ( θ ) = N ( 0 , τ 2 I ) ridge and MAP are the same estimator inverted τ 2 = σ 2 N λ \tau^2 = \dfrac{\sigma^2}{N\lambda} τ 2 = N λ σ 2 to state a tuned λ \lambda λ Gaussian mean, closed form θ ^ MAP = N N + σ 2 / τ 2 y ˉ = y ˉ 1 + λ \hat\theta_{\text{MAP}} = \dfrac{N}{N + \sigma^2/\tau^2}\bar{y} = \dfrac{\bar{y}}{1+\lambda} θ ^ MAP = N + σ 2 / τ 2 N y ˉ = 1 + λ y ˉ shrinkage toward zero
Measured across τ 2 ∈ [ 10 − 2 , 10 6 ] \tau^2 \in [10^{-2}, 10^{6}] τ 2 ∈ [ 1 0 − 2 , 1 0 6 ] ∣ ridge − MAP ∣ = 1.24 × 10 − 14 \lvert\text{ridge} - \text{MAP}\rvert = 1.24\times10^{-14} ∣ ridge − MAP ∣ = 1.24 × 1 0 − 14
prior implied penalty behaviour N ( 0 , τ 2 I ) \mathcal{N}(\mathbf{0}, \tau^2\mathbf{I}) N ( 0 , τ 2 I ) λ ∥ θ ∥ 2 \lambda\lVert\boldsymbol\theta\rVert^2 λ ∥ θ ∥ 2 smooth shrinkage Laplace λ ∥ θ ∥ 1 \lambda\lVert\boldsymbol\theta\rVert_1 λ ∥ θ ∥ 1 exact zeros uniform (improper) none recovers the MLE N ( m , τ 2 I ) \mathcal{N}(\mathbf{m}, \tau^2\mathbf{I}) N ( m , τ 2 I ) λ ∥ θ − m ∥ 2 \lambda\lVert\boldsymbol\theta - \mathbf{m}\rVert^2 λ ∥ θ − m ∥ 2 shrinks toward m \mathbf{m} m
Measured on Table 8.2: the intercept falls monotonically from 8.8074 8.8074 8.8074 rises from
2.1225 2.1225 2.1225 2.2988 2.2988 2.2988 τ 2 = 31.71 \tau^2 = 31.71 τ 2 = 31.71
case degree training risk expected risk ratio ∥ θ ∥ \lVert\boldsymbol\theta\rVert ∥ θ ∥ overfitting 11 11 11 0.005232 \mathbf{0.005232} 0.005232 3.9173 3.9173 3.9173 748.8 748.8 748.8 3616.98 3616.98 3616.98 underfitting 1 1 1 0.477771 0.477771 0.477771 0.7254 0.7254 0.7254 1.5 1.5 1.5 0.81 0.81 0.81 fitting well 4 4 4 0.113967 0.113967 0.113967 0.3806 \mathbf{0.3806} 0.3806 3.3 3.3 3.3 5.67 5.67 5.67
Overfitting has the lowest training risk of the three. Underfitting is visible from training data
alone; overfitting is not — the parameter norm is the only training-side warning.
The risk decomposition , verified to Monte Carlo error:
E [ risk ] = σ 2 ⏟ noise + E x [ ( f ˉ ( x ) − f ∗ ( x ) ) 2 ] ⏟ bias 2 + E x [ V a r ( f ( x ) ) ] ⏟ variance \mathbb{E}[\text{risk}] = \underbrace{\sigma^2}_{\text{noise}} + \underbrace{\mathbb{E}_x\!\left[(\bar{f}(x) - f^*(x))^2\right]}_{\text{bias}^2} + \underbrace{\mathbb{E}_x\!\left[\mathrm{Var}(f(x))\right]}_{\text{variance}} E [ risk ] = noise σ 2 + bias 2 E x [ ( f ˉ ( x ) − f ∗ ( x ) ) 2 ] + variance E x [ Var ( f ( x )) ] degree noise bias2 ^2 2 variance sum measured 1 1 1 0.122500 0.122500 0.122500 0.456620 \mathbf{0.456620} 0.456620 0.052142 0.052142 0.052142 0.631262 0.631262 0.631262 0.632236 0.632236 0.632236 3 3 3 0.122500 0.122500 0.122500 0.057673 0.057673 0.057673 0.050214 0.050214 0.050214 0.230387 0.230387 0.230387 0.230633 0.230633 0.230633 5 5 5 0.122500 0.122500 0.122500 0.001261 0.001261 0.001261 0.191547 0.191547 0.191547 0.315308 0.315308 0.315308 0.315533 0.315533 0.315533 9 9 9 0.122500 0.122500 0.122500 0.704671 0.704671 0.704671 2104.812958 \mathbf{2104.812958} 2104.812958 2105.640129 2105.640129 2105.640129 2105.550969 2105.550969 2105.550969
p ( y n ∣ x n , θ ) = ϕ ( θ ⊤ x n ) (Eq 8.21) p(y_n \mid \mathbf{x}_n, \boldsymbol\theta) = \phi(\boldsymbol\theta^\top\mathbf{x}_n) \qquad \text{(Eq 8.21)} p ( y n ∣ x n , θ ) = ϕ ( θ ⊤ x n ) (Eq 8.21) ϕ \phi ϕ link function ; the family is the generalized linear model (McCullagh and Nelder,
1989; Agresti, 2002). Binary classification and count data live here, via the exponential family of §6.6.
§8.4.1’s definition: a probabilistic model is specified by the joint distribution of all its random
variables. Everything else follows by a rule of probability:
what you want from the joint, by prior and likelihood the product rule — they are its factors marginal likelihood p ( x ) p(\mathbf{x}) p ( x ) the sum rule — integrate out θ \boldsymbol\theta θ posterior divide the joint by the marginal
what equation posterior p ( θ ∣ X ) = p ( X ∣ θ ) p ( θ ) p ( X ) p(\boldsymbol\theta \mid \mathcal{X}) = \dfrac{p(\mathcal{X} \mid \boldsymbol\theta)p(\boldsymbol\theta)}{p(\mathcal{X})} p ( θ ∣ X ) = p ( X ) p ( X ∣ θ ) p ( θ ) p ( X ) = ∫ p ( X ∣ θ ) p ( θ ) d θ \ p(\mathcal{X}) = \displaystyle\int p(\mathcal{X} \mid \boldsymbol\theta)p(\boldsymbol\theta)\,d\boldsymbol\theta p ( X ) = ∫ p ( X ∣ θ ) p ( θ ) d θ Eq 8.22 predictive p ( x ) = ∫ p ( x ∣ θ ) p ( θ ) d θ = E θ [ p ( x ∣ θ ) ] p(\mathbf{x}) = \displaystyle\int p(\mathbf{x} \mid \boldsymbol\theta)p(\boldsymbol\theta)\,d\boldsymbol\theta = \mathbb{E}_{\boldsymbol\theta}[p(\mathbf{x} \mid \boldsymbol\theta)] p ( x ) = ∫ p ( x ∣ θ ) p ( θ ) d θ = E θ [ p ( x ∣ θ )] Eq 8.23
Gaussian closed form. With Σ N = ( Φ ⊤ Φ / σ 2 + I / τ 2 ) − 1 \boldsymbol\Sigma_N = \left(\boldsymbol\Phi^\top\boldsymbol\Phi/\sigma^2 + \mathbf{I}/\tau^2\right)^{-1} Σ N = ( Φ ⊤ Φ / σ 2 + I / τ 2 ) − 1 m N = Σ N Φ ⊤ y / σ 2 \mathbf{m}_N = \boldsymbol\Sigma_N\boldsymbol\Phi^\top\mathbf{y}/\sigma^2 m N = Σ N Φ ⊤ y / σ 2
mean = ϕ ⊤ m N , variance = σ 2 ⏟ noise + ϕ ⊤ Σ N ϕ ⏟ not knowing θ \text{mean} = \boldsymbol\phi^\top\mathbf{m}_N, \qquad \text{variance} = \underbrace{\sigma^2}_{\text{noise}} + \underbrace{\boldsymbol\phi^\top\boldsymbol\Sigma_N\boldsymbol\phi}_{\text{not knowing }\boldsymbol\theta} mean = ϕ ⊤ m N , variance = noise σ 2 + not knowing θ ϕ ⊤ Σ N ϕ The mean is identical to the plug-in prediction. m N \mathbf{m}_N m N 8.88 × 10 − 16 8.88\times10^{-16} 8.88 × 1 0 − 16
Scalar case: 1 σ N 2 = N σ 2 + 1 τ 2 \dfrac{1}{\sigma_N^2} = \dfrac{N}{\sigma^2} + \dfrac{1}{\tau^2} σ N 2 1 = σ 2 N + τ 2 1 precisions add .
region plug-in p ( x ∣ θ ∗ ) p(x \mid \boldsymbol\theta^*) p ( x ∣ θ ∗ ) Eq 8.23 plug width full width interpolation 0.9287 0.9287 0.9287 0.9506 \mathbf{0.9506} 0.9506 1.3720 1.3720 1.3720 1.4903 1.4903 1.4903 mild extrapolation 0.5701 0.5701 0.5701 0.9501 \mathbf{0.9501} 0.9501 1.3720 1.3720 1.3720 3.8869 3.8869 3.8869 far extrapolation 0.1796 \mathbf{0.1796} 0.1796 0.9498 \mathbf{0.9498} 0.9498 1.3720 1.3720 1.3720 13.2538 13.2538 13.2538
The plug-in width is 2 × 1.96 × σ = 1.3720 2 \times 1.96 \times \sigma = 1.3720 2 × 1.96 × σ = 1.3720 everywhere , because it does not depend
on x x x
N N N largest posterior sd full / plug-in width 5 5 5 0.821958 0.821958 0.821958 1.248213 1.248213 1.248213 25 25 25 0.392692 0.392692 0.392692 1.039472 1.039472 1.039472 1000 1000 1000 0.074293 0.074293 0.074293 1.001109 1.001109 1.001109 10000 10000 10000 0.022878 0.022878 0.022878 1.000113 \mathbf{1.000113} 1.000113
Failure modes, measured. Misspecified model class (sine data, cubic model): full predictive reaches
only 0.8908 0.8908 0.8908 σ \sigma σ 0.6697 0.6697 0.6697
Approximations when the prior is not conjugate: MCMC; the Laplace approximation; variational
inference; expectation propagation.
what equation the generative conditional p ( x ∣ θ , z ) p(\mathbf{x} \mid \boldsymbol\theta, \mathbf{z}) p ( x ∣ θ , z ) Eq 8.24 the likelihood p ( x ∣ θ ) = ∫ p ( x ∣ θ , z ) p ( z ) d z p(\mathbf{x} \mid \boldsymbol\theta) = \displaystyle\int p(\mathbf{x} \mid \boldsymbol\theta, \mathbf{z})p(\mathbf{z})\,d\mathbf{z} p ( x ∣ θ ) = ∫ p ( x ∣ θ , z ) p ( z ) d z Eq 8.25 parameter posterior p ( θ ∣ X ) = p ( X ∣ θ ) p ( θ ) / p ( X ) p(\boldsymbol\theta \mid \mathcal{X}) = p(\mathcal{X} \mid \boldsymbol\theta)p(\boldsymbol\theta)/p(\mathcal{X}) p ( θ ∣ X ) = p ( X ∣ θ ) p ( θ ) / p ( X ) Eq 8.26 latent posterior p ( z ∣ X ) = p ( X ∣ z ) p ( z ) / p ( X ) p(\mathbf{z} \mid \mathcal{X}) = p(\mathcal{X} \mid \mathbf{z})p(\mathbf{z})/p(\mathcal{X}) p ( z ∣ X ) = p ( X ∣ z ) p ( z ) / p ( X ) Eq 8.27 the tractable one p ( z ∣ X , θ ) = p ( X ∣ z , θ ) p ( z ) p ( X ∣ θ ) p(\mathbf{z} \mid \mathcal{X}, \boldsymbol\theta) = \dfrac{p(\mathcal{X} \mid \mathbf{z}, \boldsymbol\theta)p(\mathbf{z})}{p(\mathcal{X} \mid \boldsymbol\theta)} p ( z ∣ X , θ ) = p ( X ∣ θ ) p ( X ∣ z , θ ) p ( z ) Eq 8.28
Equation 8.25’s rule: the likelihood must not depend on z \mathbf{z} z x \mathbf{x} x θ \boldsymbol\theta θ z \mathbf{z} z θ \boldsymbol\theta θ “not possible in
general” ; Eq 8.28 is the compromise, and it is what Chapters 10 and 11 actually compute.
EM , measured on a two-component mixture: the E step is Eq 8.28, the M step maximises given it, and the
likelihood never falls — smallest step over 40 iterations − 9.09 × 10 − 13 -9.09\times10^{-13} − 9.09 × 1 0 − 13 No guarantee about which local maximum.
what rule nodes random variables arrows conditional probabilities; a → b a \to b a → b p ( b ∣ a ) p(b \mid a) p ( b ∣ a ) shaded node observed ; unshaded is latentplate repeat everything inside N N N hyperprior a prior on a prior’s parameters; deterministic parameters lose their circle factorization → graph one node per variable; an arrow in from every variable it is conditioned on graph → factorization Eq 8.31 : p ( x ) = ∏ k = 1 K p ( x k ∣ P a k ) p(\mathbf{x}) = \prod_{k=1}^{K} p(x_k \mid \mathrm{Pa}_k) p ( x ) = ∏ k = 1 K p ( x k ∣ Pa k )
K K K full joint, 2 K − 1 2^K-1 2 K − 1 ≤ 2 \le 2 ≤ 2 ratio 5 5 5 31 31 31 15 15 15 2.067 2.067 2.067 20 20 20 1.049 × 10 6 1.049\times10^{6} 1.049 × 1 0 6 75 75 75 1.398 × 10 4 1.398\times10^{4} 1.398 × 1 0 4 100 100 100 1.268 × 10 30 \mathbf{1.268\times10^{30}} 1.268 × 1 0 30 395 \mathbf{395} 395 3.209 × 10 27 3.209\times10^{27} 3.209 × 1 0 27
p ( a ) p ( b ∣ a ) p(a)p(b \mid a) p ( a ) p ( b ∣ a ) 0.00 × 10 0 0.00\times10^{0} 0.00 × 1 0 0 p ( b ) p ( a ∣ b ) p(b)p(a \mid b) p ( b ) p ( a ∣ b ) 5.55 × 10 − 17 5.55\times10^{-17} 5.55 × 1 0 − 17 “with additional
assumptions (Pearl, 2009)” .
A ⊥ ⊥ B ∣ C A \perp\!\!\!\perp B \mid C A ⊥ ⊥ B ∣ C every trail (path ignoring arrow direction) from A A A B B B
meeting shape blocked when conditioning… head to tail u → n → v u \to n \to v u → n → v n ∈ C n \in C n ∈ C blocks tail to tail u ← n → v u \leftarrow n \to v u ← n → v n ∈ C n \in C n ∈ C blocks head to head u → n ← v u \to n \leftarrow v u → n ← v n ∉ C n \notin C n ∈ / C and no descendant of n n n C C C opens
Example 8.9 on Figure 8.11 (a → b a \to b a → b a → d a \to d a → d b → c b \to c b → c c → d c \to d c → d d → e d \to e d → e 4,000,000 4{,}000{,}000 4 , 000 , 000
eq claim book says partial correlation 8.35 b ⊥ ⊥ d ∣ a , c b \perp\!\!\!\perp d \mid a, c b ⊥ ⊥ d ∣ a , c independent − 0.000320 -0.000320 − 0.000320 8.36 a ⊥ ⊥ c ∣ b a \perp\!\!\!\perp c \mid b a ⊥ ⊥ c ∣ b independent − 0.000079 -0.000079 − 0.000079 8.37 b ⊥̸ ⊥ d ∣ c b \not\perp\!\!\!\perp d \mid c b ⊥ ⊥ d ∣ c dependent 0.272172 0.272172 0.272172 8.38 a ⊥̸ ⊥ c ∣ b , e a \not\perp\!\!\!\perp c \mid b, e a ⊥ ⊥ c ∣ b , e dependent − 0.146064 -0.146064 − 0.146064
The collider, isolated: c o r r ( a , c ) \mathrm{corr}(a,c) corr ( a , c ) − 0.000079 -0.000079 − 0.000079 { b } \{b\} { b } − 0.237939 -0.237939 − 0.237939 { b , d } \{b, d\} { b , d } − 0.146064 -0.146064 − 0.146064 { b , e } \{b, e\} { b , e } descendant is enough.
Three families (§8.5.3): directed (Bayesian networks), undirected (Markov random fields), factor
graphs. “Not every distribution can be represented in a particular choice of graphical model.”
Inner loop chooses (the validation set); outer loop reports (the test set).
600 600 600 60 60 60 no signal at all , n = 30 n = 30 n = 30
value flat CV score of the winner 0.865343 0.865343 0.865343 its true risk on fresh data 1.254179 1.254179 1.254179 chance 1.000000 1.000000 1.000000
Flat CV reports 13.5 % 13.5\% 13.5% better than chance on data containing nothing. The bias tracks how hard
you searched — with only 10 10 10 60 60 60 0.5 % 0.5\% 0.5%
what equation generative process M k ∼ p ( M ) M_k \sim p(M) M k ∼ p ( M ) θ k ∼ p ( θ ∣ M k ) \ \boldsymbol\theta_k \sim p(\boldsymbol\theta \mid M_k) θ k ∼ p ( θ ∣ M k ) D ∼ p ( D ∣ θ k ) \ \mathcal{D} \sim p(\mathcal{D} \mid \boldsymbol\theta_k) D ∼ p ( D ∣ θ k ) Eq 8.40–8.42 model posterior p ( M k ∣ D ) ∝ p ( M k ) p ( D ∣ M k ) p(M_k \mid \mathcal{D}) \propto p(M_k)\,p(\mathcal{D} \mid M_k) p ( M k ∣ D ) ∝ p ( M k ) p ( D ∣ M k ) Eq 8.43 model evidence p ( D ∣ M k ) = ∫ p ( D ∣ θ k ) p ( θ k ∣ M k ) d θ k p(\mathcal{D} \mid M_k) = \displaystyle\int p(\mathcal{D} \mid \boldsymbol\theta_k)p(\boldsymbol\theta_k \mid M_k)\,d\boldsymbol\theta_k p ( D ∣ M k ) = ∫ p ( D ∣ θ k ) p ( θ k ∣ M k ) d θ k Eq 8.44 the choice M ∗ = arg max M k p ( M k ∣ D ) M^* = \arg\max_{M_k} p(M_k \mid \mathcal{D}) M ∗ = arg max M k p ( M k ∣ D ) Eq 8.45
With a uniform p ( M k ) = 1 / K p(M_k) = 1/K p ( M k ) = 1/ K maximising the evidence .
The automatic Occam’s razor. No prior favouring simple models is needed: p ( D ∣ M ) p(\mathcal{D} \mid M) p ( D ∣ M ) over datasets , so breadth costs depth. Measured over 200,000 200{,}000 200 , 000
K L ( M 1 ∥ M 2 ) = 3.4062 nats , K L ( M 2 ∥ M 1 ) = 50.7124 nats \mathrm{KL}(M_1 \Vert M_2) = 3.4062 \text{ nats}, \qquad \mathrm{KL}(M_2 \Vert M_1) = 50.7124 \text{ nats} KL ( M 1 ∥ M 2 ) = 3.4062 nats , KL ( M 2 ∥ M 1 ) = 50.7124 nats Both positive necessarily — each model beats the other on its own data, so none can win everywhere .
That is Figure 8.14. Region C C C w = 0.571273 w = 0.571273 w = 0.571273
The evidence splits exactly (verified to 5.68 × 10 − 14 5.68\times10^{-14} 5.68 × 1 0 − 14 A = Φ ⊤ Φ / σ 2 + I / τ 2 \mathbf{A} = \boldsymbol\Phi^\top\boldsymbol\Phi/\sigma^2 + \mathbf{I}/\tau^2 A = Φ ⊤ Φ / σ 2 + I / τ 2
log p ( D ∣ M ) = log p ( D ∣ m ) ⏟ best fit − ∥ m ∥ 2 2 τ 2 − D 2 log τ 2 − 1 2 log ∣ A ∣ ⏟ Occam factor \log p(\mathcal{D} \mid M) = \underbrace{\log p(\mathcal{D} \mid \mathbf{m})}_{\text{best fit}} \underbrace{- \frac{\lVert\mathbf{m}\rVert^2}{2\tau^2} - \frac{D}{2}\log\tau^2 - \frac{1}{2}\log\lvert\mathbf{A}\rvert}_{\text{Occam factor}} log p ( D ∣ M ) = best fit log p ( D ∣ m ) Occam factor − 2 τ 2 ∥ m ∥ 2 − 2 D log τ 2 − 2 1 log ∣ A ∣ Degree 1 → 11 1 \to 11 1 → 11 45.70 45.70 45.70 12.67 12.67 12.67 The penalty is real,
automatic, and gentle.
p ( M 1 ∣ D ) p ( M 2 ∣ D ) ⏟ posterior odds = p ( M 1 ) p ( M 2 ) ⏟ prior odds p ( D ∣ M 1 ) p ( D ∣ M 2 ) ⏟ Bayes factor (Eq 8.46) \underbrace{\frac{p(M_1 \mid \mathcal{D})}{p(M_2 \mid \mathcal{D})}}_{\text{posterior odds}} = \underbrace{\frac{p(M_1)}{p(M_2)}}_{\text{prior odds}}\ \underbrace{\frac{p(\mathcal{D} \mid M_1)}{p(\mathcal{D} \mid M_2)}}_{\text{Bayes factor}} \qquad \text{(Eq 8.46)} posterior odds p ( M 2 ∣ D ) p ( M 1 ∣ D ) = prior odds p ( M 2 ) p ( M 1 ) Bayes factor p ( D ∣ M 2 ) p ( D ∣ M 1 ) (Eq 8.46) With a uniform model prior this reduces to the Bayes factor alone (Eq 8.47 ): above 1 1 1 M 1 M_1 M 1 M 2 M_2 M 2
log Bayes factor Jeffreys (1961) < log 3 < \log 3 < log 3 barely worth mentioning log 3 \log 3 log 3 log 10 \log 10 log 10 substantial log 10 \log 10 log 10 log 30 \log 30 log 30 strong log 30 \log 30 log 30 log 100 \log 100 log 100 very strong > log 100 > \log 100 > log 100 decisive
Measured on 25 points: degrees 0 0 0 2 2 2 decisive (70.83 70.83 70.83 33.28 33.28 33.28 31.50 31.50 31.50 3 3 3 11 11 11 all “barely worth mentioning” . The evidence detects underfitting sharply and overfitting
bluntly.
“The Bayes factor always favors the simpler model since the probability of the data under a complex
model with a diffuse prior will be very small” (Murphy, 2012).
With the data held fixed , sweeping only τ 2 \tau^2 τ 2 − 39.4634 -39.4634 − 39.4634 τ 2 = 10.22 \tau^2 = 10.22 τ 2 = 10.22 τ 2 = 3.71412 × 10 6 \tau^2 = \mathbf{3.71412\times10^{6}} τ 2 = 3.71412 × 1 0 6 + 46.1873 +46.1873 + 46.1873 10 14 10^{14} 1 0 14
A Bayes factor quoted without its prior is not a reproducible number.
criterion formula equation AIC log p ( x ∣ θ ) − M \log p(\mathbf{x} \mid \boldsymbol\theta) - M log p ( x ∣ θ ) − M Eq 8.48 BIC log p ( x ∣ θ ) − 1 2 M log N \log p(\mathbf{x} \mid \boldsymbol\theta) - \tfrac{1}{2}M\log N log p ( x ∣ θ ) − 2 1 M log N Eq 8.49
Choose the largest value. BIC’s penalty exceeds AIC’s whenever N > e 2 ≈ 7.39 N > e^2 \approx 7.39 N > e 2 ≈ 7.39 1 2 log N \tfrac{1}{2}\log N 2 1 log N
criterion picks, on the chapter’s 25 points maximum likelihood degree 11 \mathbf{11} 11 AIC degree 5 5 5 BIC degree 5 5 5 exact evidence degree 6 6 6 cross-validation degree 8 8 8
BIC as an approximation: mean absolute error 8.0274 8.0274 8.0274 13.4474 13.4474 13.4474 N = 25 N = 25 N = 25 N N N ranking survives; the value does not. Its derivation drops the prior
entirely — which is both why it is inaccurate and why it is immune to the Jeffreys–Lindley paradox.
quantity value page ridge vs MAP, worst disagreement 1.24 × 10 − 14 1.24\times10^{-14} 1.24 × 1 0 − 14 805 slope peak under an isotropic prior 2.2988 2.2988 2.2988 τ 2 = 31.71 \tau^2 = 31.71 τ 2 = 31.71 805 overfitting risk ratio 748.8 × 748.8\times 748.8 × 805 posterior mean vs MAP 8.88 × 10 − 16 8.88\times10^{-16} 8.88 × 1 0 − 16 806 Eq 8.23 coverage, far extrapolation 0.9498 \mathbf{0.9498} 0.9498 0.1796 \mathbf{0.1796} 0.1796 806 width ratio at N = 10000 N = 10000 N = 10000 1.000113 1.000113 1.000113 806 Eq 8.31 vs 4M samples 0.000124 0.000124 0.000124 807 Example 8.9, four claims − 0.000320 -0.000320 − 0.000320 − 0.000079 -0.000079 − 0.000079 0.272172 0.272172 0.272172 − 0.146064 -0.146064 − 0.146064 807 collider effect − 0.000079 → − 0.237939 -0.000079 \to -0.237939 − 0.000079 → − 0.237939 807 KL divergences, Figure 8.14 3.4062 3.4062 3.4062 50.7124 50.7124 50.7124 808 region C C C w = 0.571273 w = 0.571273 w = 0.571273 808 Jeffreys–Lindley crossover τ 2 = 3.71412 × 10 6 \tau^2 = 3.71412\times10^{6} τ 2 = 3.71412 × 1 0 6 808 BIC error at N = 25 N = 25 N = 25 8.0274 8.0274 8.0274 808 flat CV on pure noise 0.865343 0.865343 0.865343 1.254179 1.254179 1.254179 808 risk decomposition residual 0.000225 0.000225 0.000225 809 evidence decomposition residual 5.68 × 10 − 14 5.68\times10^{-14} 5.68 × 1 0 − 14 809 EM smallest step − 9.09 × 10 − 13 -9.09\times10^{-13} − 9.09 × 1 0 − 13 809
Empirical risk is computable; expected risk is what you want. Everything else follows from that gap.A penalty and a prior are the same object , related by λ = σ 2 / ( N τ 2 ) \lambda = \sigma^2/(N\tau^2) λ = σ 2 / ( N τ 2 ) 1.24 × 10 − 14 1.24\times10^{-14} 1.24 × 1 0 − 14 Least squares is a consequence of Gaussian noise , not a definition of fitting.Overfitting has the lowest training risk. It is invisible without held-out data or a look at ∥ θ ∥ \lVert\boldsymbol\theta\rVert ∥ θ ∥ A point estimate is not wrong, it is incomplete. The posterior mean equals the MAP point; the covariance is what was discarded.Equation 8.23 is calibrated where a plug-in interval is not — 0.9498 0.9498 0.9498 0.1796 0.1796 0.1796 Estimation is optimization; inference is integration. They converge as N N N 1.000113 1.000113 1.000113 N = 10000 N = 10000 N = 10000 A graph is a factorization , and d-separation reads independence off it — four for four on Example 8.9.Conditioning on a collider opens a path. It is the only rule that runs backwards.The Occam’s razor is automatic and gentle , and every model-selection number is hostage to a prior or to how hard you searched.
Back to the start: When Models Meet Data Overview
