The Gaussian Mixture Model

Chapter 9 fitted a function. Chapter 10 found a subspace. Chapter 11 fits a density — the third pillar — and it starts from an admission:

In practice, the Gaussian (or similarly all other distributions we encountered so far) have limited modeling capabilities.

The fix is one line of algebra. The consequences take a chapter, and the chapter warns you at the outset: “unlike other applications we discussed earlier (linear regression or PCA), we will not find a closed-form maximum likelihood solution.”

What you’ll learn

• Equations 11.1–11.4: what a mixture model is, and why both halves of Equation 11.2 are needed — measured, dropping either one breaks the density in a different way.
• Equation 11.5, the book’s own three-component mixture, checked: it integrates to $1.000000000000$.
• The mean of a mixture is a weighted mean; the variance is not a weighted variance. Measured on Equation 11.5: $\mathbb{V}[x] = 7.79$, of which the component variances supply $0.95$ and the spread of the means supplies $6.84$ — $87.80\%$ of the total.
• How far a mixture is from any single Gaussian: skewness $0.427185$, excess kurtosis $-1.294278$, and $0.330585$ nats of KL from the closest one.
• $K$ components do not mean $K$ modes. Two equal unit-variance Gaussians show one bump until their means are more than $2\sigma$ apart — the transition measured at $d = 1.0000000$.
• $K = 1$ recovers a single Gaussian exactly, to $0.0$.
• What the extra flexibility is worth: $1.388610$ nats per data point on Figure 11.1’s kind of data, for $17$ parameters against $5$.

Intuition: averaging densities, not averaging data

There is a move here that is easy to read past. A mixture does not average the random variables — it averages the densities. Those are different operations and they give different answers.

Average two Gaussian random variables and you get a Gaussian, narrower than either. Average their densities and you get something that is not Gaussian at all, and wider than either — because the disagreement between the components is itself a form of spread.

That one distinction is the whole of §11.1, and the measurement below puts a number on it: $87.80\%$ of Equation 11.5’s variance comes from the components disagreeing about where the data is, not from any component’s own width.

diagram Where the flexibility comes from, and what it costs mermaid

  flowchart TD
  G["one Gaussian
D + D(D+1)/2 parameters"] -->|"convex combination
Equations 11.1, 11.2"| M["a mixture
K times that, plus K-1"]
  M --> A["multimodal: Figure 11.1"]
  M --> B["skewed, heavy- or light-tailed"]
  M --> C["log p has a sum inside it
Equation 11.10"]
  C --> D["no closed-form maximum likelihood
which is Sections 11.2 and 11.3"]

§11.1 The definition

A mixture model describes $p(\mathbf{x})$ as a convex combination of $K$ base distributions:

$$p(\mathbf{x}) = \sum_{k=1}^{K}\pi_k p_k(\mathbf{x}) \qquad \text{(11.1)}$$ $$0 \leq \pi_k \leq 1, \qquad \sum_{k=1}^{K}\pi_k = 1 \qquad \text{(11.2)}$$

With Gaussian components, this is the Gaussian mixture model:

$$p(\mathbf{x}\mid\boldsymbol\theta) = \sum_{k=1}^{K}\pi_k\,\mathcal{N}(\mathbf{x}\mid\boldsymbol\mu_k,\boldsymbol\Sigma_k) \qquad \text{(11.3)}$$

with $\boldsymbol\theta := \{\boldsymbol\mu_k, \boldsymbol\Sigma_k, \pi_k : k = 1,\ldots,K\}$. The book’s worked instance:

$$p(x\mid\boldsymbol\theta) = 0.5\,\mathcal{N}(x\mid-2, \tfrac12) + 0.2\,\mathcal{N}(x\mid1, 2) + 0.3\,\mathcal{N}(x\mid4, 1) \qquad \text{(11.5)}$$

figure Figure 11.2, and what no single Gaussian can do about it matplotlib

The solid curve is the sum of the three dashed ones and integrates to 1.000000000000. On the right it is compared against the Gaussian with the same mean and variance — the closest single Gaussian in KL — which is still 0.330585 nats away and puts its peak where the mixture has a trough.

Both halves of Equation 11.2 do different jobs, and you need both

It is tempting to read $\sum_k\pi_k = 1$ as the whole constraint. Measured, each half fails on its own:

what is dropped	integral of the result	minimum value
neither — Equation 11.2 as written	$1.000000$	$1.7\times10^{-184}$
$\pi_k \geq 0$, using $[0.8, -0.1, 0.3]$	$\mathbf{1.000000}$	$\mathbf{-2.7\times10^{-2}}$
$\sum_k\pi_k = 1$, using $[0.5, 0.6, 0.3]$	$\mathbf{1.400000}$	$5.2\times10^{-184}$

The middle row is the instructive one: it still integrates to exactly $1$ and is still not a density, because it goes negative. Summing to one buys normalisation; non-negativity buys the rest. A convex combination is required, not merely an affine one.

The mean behaves. The variance does not.

For a mixture, $\mathbb{E}[x] = \sum_k\pi_k\mu_k$ — the operation you would guess. The variance is not $\sum_k\pi_k\sigma_k^2$. By the law of total variance, conditioning on which component was used:

$$\mathbb{V}[x] = \underbrace{\sum_k \pi_k\sigma_k^2}_{\text{within components}} + \underbrace{\sum_k \pi_k(\mu_k - \mathbb{E}[x])^2}_{\text{between components}}$$

On Equation 11.5, by numerical quadrature against the closed form:

quantity	value
$\mathbb{E}[x]$ by quadrature	$0.4000000000$
$\sum_k\pi_k\mu_k$	$0.4000000000$
$\mathbb{V}[x]$ by quadrature	$\mathbf{7.7900000000}$
within-component part, $\sum_k\pi_k\sigma_k^2$	$0.9500000000$
between-component part	$\mathbf{6.8400000000}$
their sum	$7.7900000000$ (gap $8.9\times10^{-16}$)

figure Eighty-eight percent of this mixture's variance is the components disagreeing matplotlib

The three one-standard-deviation bars on the left are short; the gaps between the triangles are not. Averaging the component variances would report 0.95 against a true 7.79 — an understatement of 87.80 percent — because it ignores every unit of spread that comes from the means being in different places.

The same decomposition explains three later results

This is not a curiosity about §11.1. The within/between split is exactly what §11.2’s updates trade against each other — page 1105 will measure a component’s variance shrinking as its mean moves onto a cluster, with the lost variance reappearing as between-component spread.

It is also the reason a GMM can fit a unimodal but heavy-tailed density: stack components at the same mean with different widths, and the between term is zero while the mixture is still not Gaussian. Measured on Equation 11.5, skewness $0.427185$ and excess kurtosis $-1.294278$ — both exactly $0$ for any Gaussian.

K components, and somewhere between 1 and K bumps

The book motivates mixtures by “multimodal data representations, i.e., they can describe datasets with multiple ‘clusters’”. True — but the count does not transfer. Two equal-weight unit-variance Gaussians at $\pm d$:

$d$	separation of the means	modes
$0.60$	$1.20\sigma$	$1$
$0.90$	$1.80\sigma$	$1$
$0.99$	$1.98\sigma$	$1$
$1.01$	$2.02\sigma$	$\mathbf{2}$
$1.50$	$3.00\sigma$	$2$

Bisected, the transition sits at $d = 1.0000000$ — the means must be more than exactly $2\sigma$ apart before a second bump appears at all. (The eighth decimal moves with the grid; the value itself is exactly $1$.)

figure Two components, and the second bump only appears past two sigma matplotlib

Below the threshold the two components are still there and still distinct as parameters; the density simply has one maximum. Counting bumps in a histogram is therefore a lower bound on K and never an estimate of it — which is why Section 11.5 sends you to cross-validation for choosing K.

And it moves with the weights. The more lopsided the mixture, the further apart the means must be:

$\pi_1$	$d$ at which the second mode appears
$0.5$	$1.000000$
$0.6$	$1.221327$
$0.7$	$1.357294$
$0.8$	$1.490338$
$0.9$	$\mathbf{1.657251}$

At exactly the threshold there is no strict maximum

At $d = 1$ the density is flat at the top to numerical precision, so a strict local-maximum test finds zero modes rather than one or two. That is not a bug in the search — it is what a degenerate critical point looks like, and it is the same phenomenon as a saddle in Chapter 7.

What the flexibility buys, and what it costs

On $1000$ points drawn from a three-component mixture in $\mathbb{R}^2$ — Figure 11.1’s kind of data:

model	parameters	log-likelihood
one fitted Gaussian	$5$	$-5111.2117$
the true mixture	$17$	$\mathbf{-3722.6020}$
difference		$1388.6097$ nats, or $\mathbf{1.388610}$ per point

$1.39$ nats per point is a factor of $e^{1.39} \approx 4.0$ in likelihood, per observation. That is the case for the chapter. And the price is stated in the same paragraph that makes the case:

$$\log p(\mathcal{X}\mid\boldsymbol\theta) = \sum_{n=1}^{N}\log\sum_{k=1}^{K}\pi_k\,\mathcal{N}(\mathbf{x}_n\mid\boldsymbol\mu_k,\boldsymbol\Sigma_k) \qquad \text{(11.10)}$$

The logarithm cannot get inside the sum. For $K = 1$ it can — Equation 11.11 — and that is exactly why Chapters 9 and 10 had closed forms and this chapter does not. Page 1102 measures what that costs.

pch.quizTag Is the model clear?

1. Equation 11.2 asks for weights in [0, 1] that sum to 1. Why not just the sum?

   A The sum alone is enough B Because a weight vector can sum to 1 and still produce a function that goes negative — measured at -2.7e-02 C Because otherwise the components are not Gaussian D To make the algebra tidier
   pch.quizShowAnswer

   B — Because a weight vector can sum to 1 and still produce a function that goes negative — measured at -2.7e-02 — Using weights [0.8, -0.1, 0.3] the result still integrates to exactly 1.000000 and is still not a density. Summing to one buys normalisation; non-negativity buys the rest. A convex combination is required, not an affine one.

2. How does a mixture's variance relate to its components' variances?

   A It is the weighted average of them B It is that, plus the weighted spread of the component means about the overall mean C It is the largest of them D It is unrelated
   pch.quizShowAnswer

   B — It is that, plus the weighted spread of the component means about the overall mean — Measured on Equation 11.5: the total is 7.79, the weighted component variances supply 0.95 and the spread of the means supplies 6.84 — 87.80 percent of the total. Averaging the component variances would understate the spread by that much. The mean, by contrast, really is the weighted mean.

3. A GMM has K = 5 components. How many modes does its density have?

   A Exactly 5 B Anywhere from 1 to 5 — two equal unit-variance components show one bump until their means are more than 2 sigma apart C At most 2 D Exactly 5 if the weights are equal
   pch.quizShowAnswer

   B — Anywhere from 1 to 5 — two equal unit-variance components show one bump until their means are more than 2 sigma apart — Measured by bisection, the transition for two equal-weight unit-variance Gaussians is at d = 1.0000000, i.e. means exactly two standard deviations apart. Unequal weights push it further: at a weight of 0.9 the threshold is 1.657251. Counting bumps in a histogram is a lower bound on K, never an estimate of it.

4. Why does Equation 11.10 not admit a closed-form maximum likelihood solution?

   A Because Gaussians have no closed form B Because the logarithm sits outside a sum over components and cannot be moved inside it C Because the data is high-dimensional D Because the weights are constrained
   pch.quizShowAnswer

   B — Because the logarithm sits outside a sum over components and cannot be moved inside it — For K = 1 the sum vanishes and the log applies directly to the Gaussian — Equation 11.11 — which is what gave Chapters 9 and 10 their closed forms. With K greater than 1 the log of a sum has no such simplification, and Sections 11.2 and 11.3 are the consequence.

Exercises

Exercise 1 – Break Equation 11.2, two ways

pch.exerciseTag pch.exerciseTitle

Exercise 2 – The law of total variance, on Equation 11.5

pch.exerciseTag pch.exerciseTitle

Exercise 3 – How far from Gaussian?

pch.exerciseTag pch.exerciseTitle

Exercise 4 – Find the threshold where the second bump appears

pch.exerciseTag pch.exerciseTitle

Exercise 5 – What the extra parameters are worth

pch.exerciseTag pch.exerciseTitle

Recall card

• Chapter 11 estimates a density, the third of the book’s pillars, and starts by admitting that a single Gaussian is often a poor model.
• A mixture is a convex combination of simple densities, and both halves of Equation 11.2 are needed: weights that sum to one but go negative still integrate to one and are still not a density.
• The mean of a mixture is the weighted mean of the component means. The variance is not the weighted average of the component variances.
• On the book’s own Equation 11.5 the total variance is 7.79, of which the component widths supply 0.95 and the spread of the means supplies 6.84 — 87.80 percent.
• That decomposition is the law of total variance, and it comes back in Section 11.2 when a component’s width shrinks as its mean moves.
• A mixture is not a Gaussian: measured skewness 0.427185 and excess kurtosis minus 1.294278, both exactly zero for any Gaussian.
• The closest single Gaussian is still 0.330585 nats away in KL, and puts its peak where the mixture has a trough.
• K components give between one and K modes. Two equal unit-variance Gaussians show one bump until their means are more than two standard deviations apart.
• Measured by bisection, that transition is at d = 1.0000000, and unequal weights push it out to 1.657251 at a weight of nine tenths.
• So counting bumps in a histogram is a lower bound on K and never an estimate of it.
• The flexibility is worth 1.388610 nats per data point on the kind of data Figure 11.1 shows — a factor of four in likelihood — for twelve extra parameters.
• And it costs the closed form. In Equation 11.10 the logarithm sits outside a sum over components and cannot be moved inside, which is the whole reason Sections 11.2 and 11.3 exist.

Next: Maximum Likelihood and Its Obstacle — §11.2’s opening, and exactly what the log-of-a-sum costs.