Chapter 11 Formula Sheet

One page for the whole chapter. Equation numbers are the book’s; the measured columns are this module’s, computed on the chapter’s own seven-point example $\mathcal{X} = \{-3, -2.5, -1, 0, 2, 4, 5\}$ with $K = 3$ unless stated otherwise.

The chapter in one table

Three parameters, one derivation run three times.

	§11.2.2 the means	§11.2.3 the covariances	§11.2.4 the weights
theorem	11.1	11.2	11.3
the update	$\boldsymbol\mu_k = \frac{1}{N_k}\sum_n r_{nk}\mathbf{x}_n$	$\boldsymbol\Sigma_k = \frac{1}{N_k}\sum_n r_{nk}(\mathbf{x}_n-\boldsymbol\mu_k)(\mathbf{x}_n-\boldsymbol\mu_k)^\top$	$\pi_k = \frac{N_k}{N}$
equation	11.20	11.30	11.42
what cancels	$\boldsymbol\Sigma_k^{-1}$, on the right	$\boldsymbol\Sigma_k^{-1}$, on both sides	nothing
needs a multiplier	no	no	yes, $\lambda = -N$
reads the data values	yes	yes	no
nats gained on Example 11.3–11.5	$12.32$	$1.46$	$0.14$
all three are	responsibility-weighted averages, and none is a closed form — each mentions every other parameter through $r_{nk}$

§11.1 The Gaussian mixture model

what	equation
a mixture density	$p(\mathbf{x}) = \sum_{k=1}^{K}\pi_k p_k(\mathbf{x})$ — 11.1
the weights	$0 \leq \pi_k \leq 1$, $\sum_k \pi_k = 1$ — 11.2, both halves needed
the GMM	$p(\mathbf{x}\mid\boldsymbol\theta) = \sum_k \pi_k\,\mathcal{N}(\mathbf{x}\mid\boldsymbol\mu_k,\boldsymbol\Sigma_k)$ — 11.3
its weight constraints	$0 \leq \pi_k \leq 1$, $\sum_k \pi_k = 1$ — 11.4, with $\boldsymbol\theta := \{\boldsymbol\mu_k,\boldsymbol\Sigma_k,\pi_k\}$ named in the text beneath it
the running example	$0.5\,\mathcal{N}(x\mid-2,\tfrac12) + 0.2\,\mathcal{N}(x\mid1,2) + 0.3\,\mathcal{N}(x\mid4,1)$ — 11.5
the mean	$\mathbb{E}[x] = \sum_k \pi_k\mu_k$ — a weighted average
the variance	$\mathbb{V}[x] = \sum_k \pi_k\sigma_k^2 + \sum_k \pi_k(\mu_k-\mathbb{E}[x])^2$ — not a weighted average

The variance decomposition is the law of total variance, and it returns twice

On Equation 11.5 the total is $7.79$: component widths supply $0.95$ and the spread of the means supplies $6.84$, or $87.80\%$. Page 1105 shows this quantity is conserved by every M-step, so EM only ever moves spread between the two terms.

§11.2 Maximum likelihood and the obstacle

what	equation
the likelihood	$p(\mathcal{X}\mid\boldsymbol\theta) = \prod_n p(\mathbf{x}_n\mid\boldsymbol\theta)$ — 11.9
the log-likelihood	$L = \sum_{n}\log\sum_{k}\pi_k\,\mathcal{N}(\mathbf{x}_n\mid\boldsymbol\mu_k,\boldsymbol\Sigma_k)$ — 11.10
the $K=1$ case	the log lands on the Gaussian and the answer is closed-form — 11.11
the stationarity conditions	$\partial L/\partial\boldsymbol\mu_k = \mathbf{0}$, $\partial L/\partial\boldsymbol\Sigma_k = \mathbf{0}$, $\partial L/\partial\pi_k = 0$ — 11.12–11.14
the chain rule	$\frac{\partial\log p(\mathbf{x}_n\mid\boldsymbol\theta)}{\partial\boldsymbol\theta} = \frac{1}{p(\mathbf{x}_n\mid\boldsymbol\theta)}\frac{\partial p(\mathbf{x}_n\mid\boldsymbol\theta)}{\partial\boldsymbol\theta}$ — 11.15
the factor it drags in	$\frac{1}{p(\mathbf{x}_n\mid\boldsymbol\theta)} = \big(\sum_j\pi_j\mathcal{N}(\mathbf{x}_n\mid\boldsymbol\mu_j,\boldsymbol\Sigma_j)\big)^{-1}$ — 11.16

The maximum likelihood estimate does not exist

Pin a mean to a data point and let its variance shrink: Equation 11.10 reaches $+16.377668$ at $\sigma^2 = 10^{-30}$ and has no upper bound. Of $400$ restarts on the book’s seven points, $\mathbf{218}$ — $54.5\%$ — collapse a component, and every one reports a higher likelihood than every genuine fit.

Among the honest fits there are exactly two optima, and the book’s own initialisation reaches the worse one: $-13.973323$ against an available $-13.906162$.

§11.5 names this exactly — “this happens when the mean of a mixture component is identical to a data point and the covariance tends to $0$” — and points at a Bayesian prior on the parameters. Page 1109 prices the alternative: a floor caps the singular branch, and each decade of floor is worth $\tfrac12\ln 10 = 1.1513$ nats.

§11.2.1 Responsibilities

what	equation
the responsibility	$r_{nk} := \dfrac{\pi_k\mathcal{N}(\mathbf{x}_n\mid\boldsymbol\mu_k,\boldsymbol\Sigma_k)}{\sum_j\pi_j\mathcal{N}(\mathbf{x}_n\mid\boldsymbol\mu_j,\boldsymbol\Sigma_j)}$ — 11.17
each row	$\mathbf{r}_n = [r_{n1},\ldots,r_{nK}]^\top$, a probability vector — 11.18
the book’s matrix	$\mathbf{R}$ for the seven points — 11.19
the effective count	$N_k := \sum_n r_{nk}$, fractional, $\sum_k N_k = N$ — 11.24
the data-side probability vector	$\mathbf{r}_k$ with entries $r_{nk}/N_k$ — 11.25
as a softmax	$r_{nk} = e^{-E_{nk}}/\sum_j e^{-E_{nj}}$, $E_{nk} = \tfrac12(\mathbf{x}_n-\boldsymbol\mu_k)^\top\boldsymbol\Sigma_k^{-1}(\mathbf{x}_n-\boldsymbol\mu_k) + \tfrac12\log\det\boldsymbol\Sigma_k - \log\pi_k$

Equation 11.17 as written returns NaN, and one entry of 11.19 is wrong

Typed out literally, Equation 11.17 divides a vanishing number by a vanishing number: with small variances it returns NaN for all $21$ entries. In log-space it returns the exact answer, and where both work they agree to $2.2\times10^{-16}$.

The book’s printed Equation 11.19 disagrees with the arithmetic at one entry. At $x = 0$ it prints $0.001$ and $0.999$; the computed values are $\mathbf{0.000150}$ and $0.999844$. The largest disagreement across the matrix is $0.001$.

§11.2.2 Updating the means

what	equation
the Gaussian’s $\boldsymbol\mu$-gradient	$\frac{\partial\mathcal{N}}{\partial\boldsymbol\mu_k} = (\mathbf{x}_n-\boldsymbol\mu_k)^\top\boldsymbol\Sigma_k^{-1}\mathcal{N}$ — 11.21b
assembled	$\frac{\partial L}{\partial\boldsymbol\mu_k} = \sum_n r_{nk}(\mathbf{x}_n-\boldsymbol\mu_k)^\top\boldsymbol\Sigma_k^{-1}$ — 11.22c
setting it to zero	$\sum_n r_{nk}\mathbf{x}_n = N_k\boldsymbol\mu_k^{\text{new}}$ — 11.23
Theorem 11.1	$\boldsymbol\mu_k^{\text{new}} = \dfrac{\sum_n r_{nk}\mathbf{x}_n}{N_k}$ — 11.20
as an expectation	$\boldsymbol\mu_k^{\text{new}} = \mathbb{E}_{\mathbf{r}_k}[\mathcal{X}]$ — 11.26
Example 11.3	$-4 \to -2.701230$, $0 \to -0.403411$, $8 \to 3.704287$ — 11.27–11.29

Two facts the book does not state

The update cannot leave the data’s range. It is a convex combination of the data whenever $N_k > 0$, so $\min_n x_n \le \mu_k^{\text{new}} \le \max_n x_n$. Over $4{,}000$ starting values, zero excursions — which makes it a sanity check on the responsibilities.

The $N_k$-weighted sum of the new means is the sum of the data, exactly:

$$\sum_{k}N_k\boldsymbol\mu_k^{\text{new}} = \sum_k\sum_n r_{nk}\mathbf{x}_n = \sum_n \mathbf{x}_n\underbrace{\sum_k r_{nk}}_{=\,1} = \sum_n \mathbf{x}_n$$

Measured over $2{,}000$ arbitrary responsibility matrices: worst deviation $3.6\times10^{-15}$. So the mixture’s mean is pinned to the sample mean after every M-step.

§11.2.3 Updating the covariances

what	equation
the Gaussian’s $\boldsymbol\Sigma$-gradient	$\frac{\partial p}{\partial\boldsymbol\Sigma_k} = \pi_k\mathcal{N}\cdot\left(-\tfrac12\big(\boldsymbol\Sigma_k^{-1}-\boldsymbol\Sigma_k^{-1}(\mathbf{x}_n-\boldsymbol\mu_k)(\mathbf{x}_n-\boldsymbol\mu_k)^\top\boldsymbol\Sigma_k^{-1}\big)\right)$ — 11.35
stationarity	$N_k\boldsymbol\Sigma_k^{-1} = \boldsymbol\Sigma_k^{-1}\big(\sum_n r_{nk}(\mathbf{x}_n-\boldsymbol\mu_k)(\mathbf{x}_n-\boldsymbol\mu_k)^\top\big)\boldsymbol\Sigma_k^{-1}$ — 11.37a
Theorem 11.2	$\boldsymbol\Sigma_k^{\text{new}} = \dfrac{1}{N_k}\sum_n r_{nk}(\mathbf{x}_n-\boldsymbol\mu_k)(\mathbf{x}_n-\boldsymbol\mu_k)^\top$ — 11.30
as an expectation	an importance-weighted mean of $\widetilde{\mathcal{X}}_k = \{\mathbf{x}_n-\boldsymbol\mu_k\}$ — 11.38
Example 11.4	$1 \to 0.14$, $0.2 \to 0.44$, $3 \to 1.53$ — 11.39–11.41

Equation 11.30 does not say which mean it uses, and it matters

$\boldsymbol\mu$ used	result
the old means	$1.83$, $0.60$, $19.98$
the just-updated means	$\mathbf{0.14}$, $\mathbf{0.44}$, $\mathbf{1.53}$ — the book’s Example 11.4

The M-step’s ordering is the answer: §11.3 updates the means first. Stale means measure spread about a point the component has already left, and on the third component that is a factor of $13.0878$.

An invariant stronger than the mean’s

After any M-step the mixture’s variance equals the sample variance, exactly:

$$\sum_k \pi_k^{\text{new}}\left((\sigma_k^{\text{new}})^2 + (\mu_k^{\text{new}})^2\right) = \frac{1}{N}\sum_n x_n^2$$

Measured to $3.6\times10^{-15}$ over $20{,}000$ arbitrary responsibility matrices, for every $K$ from $2$ to $10$. EM cannot make the model wider or narrower than the data — it only decides how much spread is between components and how much is within them. On the book’s example the total drops to $8.336735$ at the first M-step and never moves again.

§11.2.4 Updating the mixture weights

what	equation
the Lagrangian	$\mathfrak{L} = L + \lambda\big(\sum_k\pi_k - 1\big)$ — 11.43a
the derivative	$\frac{\partial\mathfrak{L}}{\partial\pi_k} = \frac{N_k}{\pi_k} + \lambda$ — 11.44b
the constraint	$\frac{\partial\mathfrak{L}}{\partial\lambda} = \sum_k\pi_k - 1 = 0$ — 11.45
solving	$\pi_k = -N_k/\lambda$ — 11.46, then $\sum_k N_k = N$ — 11.47
the multiplier	$\lambda = -N$ — 11.48
Theorem 11.3	$\pi_k^{\text{new}} = \dfrac{N_k}{N}$ — 11.42
Example 11.5	$\tfrac13 \to 0.293890,\ 0.287001,\ 0.419109$ — the book’s $0.29$, $0.29$, $0.42$

Why this one needs a multiplier and the other two do not

$\partial L/\partial\pi_k = \sum_n \mathcal{N}(\mathbf{x}_n\mid\boldsymbol\mu_k,\boldsymbol\Sigma_k)\big/\sum_j\pi_j\mathcal{N}(\mathbf{x}_n\mid\boldsymbol\mu_j,\boldsymbol\Sigma_j) > 0$ in every coordinate, so unconstrained ascent has nothing to converge to — measured, the weights’ sum reaches $2.54$ in six steps and keeps climbing. The means and covariances are unconstrained; the weights live on a simplex.

Contrast §10.2’s Lagrangian, where the multiplier was the answer — an eigenvalue, equal to the variance retained. Here $\lambda = -N$ is the normalising constant and carries no information.

§11.3 The EM algorithm

step	equation
initialise	$\boldsymbol\mu_k,\boldsymbol\Sigma_k,\pi_k$
E-step	$r_{nk} = \dfrac{\pi_k\mathcal{N}(\mathbf{x}_n\mid\boldsymbol\mu_k,\boldsymbol\Sigma_k)}{\sum_j\pi_j\mathcal{N}(\mathbf{x}_n\mid\boldsymbol\mu_j,\boldsymbol\Sigma_j)}$ — 11.53
M-step, in this order	$\boldsymbol\mu_k = \frac{1}{N_k}\sum_n r_{nk}\mathbf{x}_n$ — 11.54
	$\boldsymbol\Sigma_k = \frac{1}{N_k}\sum_n r_{nk}(\mathbf{x}_n-\boldsymbol\mu_k)(\mathbf{x}_n-\boldsymbol\mu_k)^\top$ — 11.55
	$\pi_k = N_k/N$ — 11.56
the guarantee	$L$ is non-decreasing at every step (Neal and Hinton, 1999)

measured on the book’s example	value
negative log-likelihood, start to first cycle	$28.325536 \to 14.410485$ — the book’s $28.3$ and $14.4$
Example 11.6’s “after five iterations”	the $10^{-6}$ criterion, reproduced exactly; $8$ at $10^{-9}$
$L$ settles before the parameters do	$5$ against $11$ iterations at $10^{-6}$

What the monotonicity guarantee looks like in float64

Over $180{,}000$ steps, $\mathbf{11{,}163}$ register a negative change in $L$. None is larger than $8.9\times10^{-15}$, and the median decrease is $1.776\times10^{-15}$ — exactly one unit in the last place of a log-likelihood near $-14$.

The theorem is about real numbers. A convergence test must therefore compare $|\Delta L|$ against a tolerance, never against zero.

Overlap sets the rate, and the initialisation sets the tail

EM converges linearly, and the constant is the ambiguity of the responsibilities: $1045$ iterations when $53\%$ of points are ambiguous against $3$ when none are — a factor of $\mathbf{348}$. When $r_{nk} \approx \tfrac1K$ the E-step barely moves, so the M-step barely moves.

A K-means initialisation cuts the median from $58$ iterations to $19$, removes the long tail, and takes the share of runs reaching the best optimum from $96\%$ to $100\%$. It does not make the optimum global — nothing can outrun an unbounded objective.

§11.4 The latent variable perspective

what	equation
the indicator	$\mathbf{z} = [z_1,\ldots,z_K]^\top$, one-hot — defined in the text of §11.4.1, unnumbered
the conditional	$p(\mathbf{x}\mid z_k=1) = \mathcal{N}(\mathbf{x}\mid\boldsymbol\mu_k,\boldsymbol\Sigma_k)$ — 11.58
the prior	$p(\mathbf{z}) = \boldsymbol\pi$, $\pi_k = p(z_k=1)$ — 11.59, 11.60
the joint, one component	$p(\mathbf{x},z_k=1) = \pi_k\mathcal{N}(\mathbf{x}\mid\boldsymbol\mu_k,\boldsymbol\Sigma_k)$ — 11.61
the joint, stacked	the $K$-vector of those products — 11.62, “which fully specifies the probabilistic model”
marginalising $\mathbf{z}$	$p(\mathbf{x}\mid\boldsymbol\theta) = \sum_{\mathbf{z}}p(\mathbf{x}\mid\boldsymbol\theta,\mathbf{z})p(\mathbf{z}\mid\boldsymbol\theta)$, recovering Equation 11.3 — 11.63, 11.65, 11.66b
the dataset likelihood	$p(\mathcal{X}\mid\boldsymbol\theta) = \prod_n\sum_k\pi_k\mathcal{N}(\mathbf{x}_n\mid\boldsymbol\mu_k,\boldsymbol\Sigma_k)$ — 11.67, i.e. Equation 11.9 again
the posterior	$p(z_k=1\mid\mathbf{x}) = \dfrac{\pi_k\mathcal{N}(\mathbf{x}\mid\boldsymbol\mu_k,\boldsymbol\Sigma_k)}{\sum_j\pi_j\mathcal{N}(\mathbf{x}\mid\boldsymbol\mu_j,\boldsymbol\Sigma_j)}$ — 11.69
the conditional factorises	$p(\mathbf{x}_1,\ldots,\mathbf{x}_N\mid\mathbf{z}_1,\ldots,\mathbf{z}_N) = \prod_n p(\mathbf{x}_n\mid\mathbf{z}_n)$ — 11.71
and over a dataset	$p(z_{nk}=1\mid\mathbf{x}_n) = r_{nk}$ — 11.72b, the identification the chapter was built toward
the E-step’s object	$Q(\boldsymbol\theta\mid\boldsymbol\theta^{(t)}) = \mathbb{E}_{\mathbf{z}\mid\mathbf{x},\boldsymbol\theta^{(t)}}[\log p(\mathbf{x},\mathbf{z}\mid\boldsymbol\theta)]$ — 11.73a
the M-step	$\boldsymbol\theta^{(t+1)} = \arg\max_{\boldsymbol\theta} Q(\boldsymbol\theta\mid\boldsymbol\theta^{(t)})$

What §11.4 buys that §11.2 cannot

The responsibility is a posterior, exactly. Computing the joint over the marginal reproduces Equation 11.17 to $0.0$ — not approximately. That is what licenses summing responsibilities as counts, differentiating a soft assignment, and alternating at all.

The M-step is the exact argmax of $Q$, not a heuristic: a forty-restart Nelder–Mead search matches it to $5.3\times10^{-15}$, with every parameter agreeing to six decimals.

$L = Q + H(\mathbf{r})$, measured to $1.1\times10^{-14}$, so $Q$ is a lower bound that is tight at the current parameters — and raising a tight lower bound cannot lower what it bounds, which is §11.3’s guarantee with a reason attached. $L$ rises by more than $Q$ does, $13.915050$ against $13.480070$, because the new responsibilities are sharper and the entropy grows too.

§11.5 What the chapter assumes, and what it costs

§11.5 is a page of caveats. Page 1109 runs each one.

what §11.5 says	what page 1109 measured
”$K$ is known… in practice this is often not the case”; use nested cross-validation	held-out $L$, BIC and AIC pick $3$, $3$ and $4$ on the same table
— the same, as $N$ grows	BIC reaches $20/20$; held-out $L$ peaks at $19/20$ and falls back to $14/20$
— the same, on non-Gaussian data	a uniform asks for $5$ or $6$, a Student-$t$ for $3$ that mean nothing
“the likelihood approaches infinity” when a component collapses	a floor prices that at $1.1513$ nats per decade; the honest optimum is $-128.9013$
a Bayesian prior would give “a principled way to determine the number of mixture components”	BIC, its asymptotic approximation, is the only criterion here that stays right as $N$ grows
histograms and kernel density estimation as the non-parametric route	$600$ stored points, $13.7$ nats behind $8$ numbers, and the bandwidth swings the score $273$
(not mentioned) the covariance structure of Equation 11.3	tying costs $1.8769$ nats held out; BIC then prefers the tied model
(not mentioned) identifiability	every optimum has $K!$ copies, identical to a spread of $0.000\times10^{0}$

Every measured constant in this chapter

quantity	value	page
variance of Equation 11.5, within / between	$0.95$ / $6.84$ ($87.80\%$) of $7.79$	1101
skewness, excess kurtosis of Equation 11.5	$0.427185$, $-1.294278$	1101
KL to the closest single Gaussian	$0.330585$ nats	1101
two equal Gaussians merge at	$d = 1.0000000$ ($2\sigma$ apart); $1.657251$ at weight $0.9$	1101
a GMM’s edge over one Gaussian	$1.388610$ nats per point, for $12$ extra parameters	1101
the book’s starting negative log-likelihood	$28.325536$ — its $28.3$	1102
Equation 11.9’s product underflows at	about $200$ data points	1102
$K=1$ closed form vs an optimiser	$6\times10^{-8}$	1102
analytic gradient vs central differences	$3\times10^{-10}$	1102
feeding Equation 11.20 back in	moves the means $4.295713$, then $0.130775$, then $0.018414$	1102
$400$ restarts on $7$ points	$17$ distinct optima; $218$ ($54.5\%$) collapse	1102
the unbounded likelihood	$+16.377668$ at $\sigma^2 = 10^{-30}$	1102
the book’s basin vs the best honest one	$-13.973323$ vs $-13.906162$	1102
Equation 11.19’s disputed entry ($x=0$)	printed $0.001$; computed $\mathbf{0.000150}$	1103
responsibilities as a softmax of $-\mathbf{E}$	$1.1\times10^{-16}$	1103
mean row entropy at convergence	$0.014$ against a possible $1.099$	1103
Equation 11.17 typed as written	$21$ NaNs of $21$; log-space agrees to $2.2\times10^{-16}$	1103
K-means from the same start	centres $1.504119$ apart, split $3$-$2$-$2$	1103
Example 11.3 reproduced	$-2.701230$, $-0.403411$, $3.704287$	1104
means stay inside the data’s range	$0$ excursions over $4{,}000$ starts	1104
$\sum_k N_k\boldsymbol\mu_k = \sum_n \mathbf{x}_n$	$3.6\times10^{-15}$ over $2{,}000$ matrices	1104
component death, $N_k$ underflows at	$\mu_3 = 73.585070$ (and $-69.812512$)	1104
the mean update alone	$12.321385$ nats, stalling at $-15.964728$	1104
Equation 11.30 with old vs new means	$1.83/0.60/19.98$ vs $0.14/0.44/1.53$ — factor $13.0878$	1105
positive semi-definiteness	$0$ negatives in $60{,}000$ values, no repair step	1105
the mixture’s variance after any M-step	$=$ the sample variance, $3.6\times10^{-15}$ over $20{,}000$	1105
that total on the book’s example	$8.336735$ from the first M-step onward	1105
the singularity from the variance side	$2.5\times10^{-4}$ at a responsibility of $0.99999$	1105
unconstrained ascent on the weights	the sum reaches $2.54$ in six steps	1106
the Lagrange multiplier	exactly $-N$, deviation $0$	1106
Example 11.5 reproduced	$0.293890$, $0.287001$, $0.419109$	1106
one full cycle	$28.325536 \to 14.410485$ — the book’s $28.3 \to 14.4$	1106
the split of the $13.92$ nats	means $12.32$, variances $1.46$, weights $0.14$	1106
“after five iterations”	the $10^{-6}$ criterion exactly; $8$ at $10^{-9}$	1107
$L$ vs parameters settling	$5$ against $11$ iterations at $10^{-6}$	1107
monotonicity over $180{,}000$ steps	$11{,}163$ negative, none above $8.9\times10^{-15}$	1107
median decrease	$1.776\times10^{-15}$ — one ULP	1107
overlap and the convergence rate	$1045$ iterations at $53\%$ ambiguous vs $3$ at $0\%$	1107
K-means initialisation	median $58 \to 19$; best optimum $96\% \to 100\%$	1107
Equation 11.69 vs Equation 11.17	$0.0$	1108
the M-step as the argmax of $Q$	$5.3\times10^{-15}$ over $40$ restarts	1108
$L = Q + H(\mathbf{r})$	$1.1\times10^{-14}$	1108
the step’s rise in $L$ and in $Q$	$13.915050$ against $13.480070$	1108
parameter recovery, $N = 200$ to $200{,}000$	errors fall by factors of $10$, $20$ and $11$	1108
choosing $K$: train / held-out / BIC / AIC	$6$ / $\mathbf{3}$ / $\mathbf{3}$ / $4$	1109
a variance floor’s price	$\tfrac12\ln 10 = 1.1513$ nats per decade	1109
the honest six-component optimum	$-128.9013$, against $-120.9761$ for a spike	1109
label switching, all $3!$ relabellings	spread $0.000\times10^{0}$	1109
GMM as a classifier, held out	$0.9650$ — equal to the labelled fit, Bayes rate $0.9633$	1109
tying the variances	costs $1.8769$ nats held out, saves $8.8856$ of BIC	1109
held-out $L$ recovering $K$ as $N$ grows	$11 \to 19 \to 19 \to 15 \to 14$ of $20$	1109
BIC recovering $K$ as $N$ grows	$10 \to 13 \to 20 \to 20 \to 20$ of $20$	1109
GMM vs the best KDE	$8$ numbers, $13.7$ nats ahead of $600$ points	1109

The chapter in twelve lines

1. A mixture buys multimodality for a handful of parameters — $1.388610$ nats per point for twelve of them — and pays with the closed form.
2. Equation 11.2 needs both halves. Weights that sum to one but go negative still integrate to one and are still not a density.
3. The variance of a mixture is not the average of the variances, and that decomposition is conserved by every M-step.
4. Equation 11.10’s log sits outside a sum, so the chain rule drags $1/p(\mathbf{x}_n)$ into every derivative and every parameter’s equation mentions every other.
5. Equation 11.20 is a fixed point, not a formula. Feeding it back in moves the means by $4.295713$.
6. The responsibility is that dragged-in factor, named — and §11.4 shows it is a posterior, exactly.
7. All three updates are the same derivation: weighted average of the data, of the squared deviations, of nothing at all.
8. Only the weights need a Lagrange multiplier, because only they are constrained — and it comes out as $-N$, carrying no information.
9. Order the M-step. Equation 11.30 with stale means is wrong by a factor of $13.0878$ on the book’s third component.
10. EM’s guarantee holds to one ULP, so test convergence against a tolerance and never against zero.
11. The maximum likelihood estimate does not exist. A floor prices the singularity at $1.1513$ nats per decade; a prior removes it.
12. §11.5’s ”$K$ is known” is doing the heaviest lifting in the chapter — held-out likelihood is not even a consistent estimator of it.

Back to the start: Density Estimation Overview