Maximum Likelihood Estimation

§8.2 never wrote down a probability. This section does, and the book is explicit about the mapping:

In Section 8.3.1, we introduce the likelihood, which is analogous to the concept of loss functions (Section 8.2.2) in empirical risk minimization. The concept of priors (Section 8.3.2) is analogous to the concept of regularization (Section 8.2.3).

This page does the first correspondence and page 805 does the second. And “analogous” turns out to be an understatement — measured, the two objectives are affine transforms of each other, so they rank every parameter identically.

What you’ll learn

• Equation 8.14, the negative log-likelihood, and why the negative sign is there at all.
• The interpretive point the book makes twice: $p(\mathbf{x} \mid \boldsymbol\theta)$ read two ways is two different objects.
• Measured: the likelihood does not integrate to 1 over $\boldsymbol\theta$ — it integrates to $0.104496$ in the worked case. It is not a distribution over parameters.
• Equations 8.15–8.17: the supervised setting, and where the independence assumption turns a product into a sum.
• Example 8.5, Equation 8.18: Gaussian maximum likelihood is least squares. Measured, the minimisers agree to $1.8\times10^{-15}$ and the objectives differ by $\mathcal{L} = 102.040816\,R_{\text{emp}} - 3.272090$.
• The three asymptotic properties of the MLE, and the warning attached to them: measured, $N \times$ error is roughly constant, confirming $1/N$ decay.

Intuition: which knob best explains what you saw?

You have a machine with a dial on it. Turn the dial to $\boldsymbol\theta$ and the machine spits out numbers according to $p(\mathbf{x} \mid \boldsymbol\theta)$. You did not see the dial, but you saw the numbers.

Maximum likelihood asks: which dial setting makes the numbers I saw least surprising? Not “which is most probable” — the dial has no probability attached, that would need a prior — but which setting assigns the highest density to the observations.

The whole trick is that $p(\mathbf{x} \mid \boldsymbol\theta)$ can be read in two directions. Read it with $\boldsymbol\theta$ nailed down and $\mathbf{x}$ free, it is a distribution over data. Read it with $\mathbf{x}$ nailed down — because you observed it — and $\boldsymbol\theta$ free, it is the likelihood. Same formula. Different function, different variable, different meaning.

diagram Section 8.3.1, and where it lands mermaid

  flowchart TD
  M["a probabilistic model
p(y | x, theta)"]
  M -->|"fix theta, vary y"| D["a DISTRIBUTION over data
integrates to 1"]
  M -->|"fix y, vary theta"| L["the LIKELIHOOD
does NOT integrate to 1"]
  L --> NLL["Eq 8.14: negative log-likelihood
minimise instead of maximise"]
  NLL -->|"independence, Eq 8.17"| SUM["a SUM over examples
instead of a product"]
  SUM -->|"Gaussian, Eq 8.18"| LS["exactly least squares
Section 8.2's Example 8.2"]
  L -.->|"multiply by a prior"| POST["the POSTERIOR
Section 8.3.2"]
  style POST stroke-dasharray: 4 3

Equation 8.14: the negative log-likelihood

For data represented by a random variable $\mathbf{x}$ and a family of densities $p(\mathbf{x} \mid \boldsymbol\theta)$:

$$\mathcal{L}_{\mathbf{x}}(\boldsymbol\theta) = -\log p(\mathbf{x} \mid \boldsymbol\theta)$$

The subscript emphasises that $\boldsymbol\theta$ is varying and $\mathbf{x}$ is fixed, and the book notes it is usually dropped, leaving $\mathcal{L}(\boldsymbol\theta)$ — “as it is really a function of $\boldsymbol\theta$”.

Three transformations happened in that one line, and each has a reason:

• The logarithm. Independence makes the joint density a product over examples, and a product of $N$ small numbers underflows. §0.3 measured the failure: a product of $162$ probabilities of $0.01$ reaches exactly 0.0 in float64. The logarithm turns it into a sum, which does not.
• The negative sign. The book calls it “a historical artifact that is due to the convention that we want to maximize likelihood, but numerical optimization literature tends to study minimization of functions.” That is all it is.
• Nothing else. $\log$ is strictly increasing and negation reverses order exactly once, so $\arg\max_{\boldsymbol\theta} p = \arg\min_{\boldsymbol\theta} \mathcal{L}$ with no approximation.

The two readings

The book states this carefully, so it is worth quoting both halves.

Fix $\boldsymbol\theta$, vary $\mathbf{x}$: “It is a distribution that models the uncertainty of the data. In other words, once we have chosen the type of function we want as a predictor, the likelihood provides the probability of observing data $\mathbf{x}$.”

Fix $\mathbf{x}$, vary $\boldsymbol\theta$: “It tells us how likely a particular setting of $\boldsymbol\theta$ is for the observations $\mathbf{x}$. Based on this second view, the maximum likelihood estimator gives us the most likely parameter $\boldsymbol\theta$ for the set of data.”

The likelihood is not a probability distribution over the parameters

This is the one that causes real confusion, so here it is measured. Take three observations $(1.2, 2.1, 1.7)$ from $\mathcal{N}(\theta, 0.8^2)$ and integrate:

what	over what	integral
$p(y \mid \theta = 2)$	$y$	$1.000000$
$\prod_n p(y_n \mid \theta)$	$\theta$	$\mathbf{0.104496}$

The first is a density and integrates to $1$ by definition. The second does not, and there is no reason it should — it was never normalised in $\boldsymbol\theta$, and nothing requires it to be finite.

So “the most likely $\boldsymbol\theta$” is a slight abuse of language: the likelihood is not a probability of $\boldsymbol\theta$. Turning it into one requires a prior, which is §8.3.2.

Equations 8.15–8.17: the supervised setting

Given pairs $(\mathbf{x}_1, y_1), \dots, (\mathbf{x}_N, y_N)$ with $\mathbf{x}_n \in \mathbb{R}^D$ and $y_n \in \mathbb{R}$, we specify the conditional distribution of the labels given the examples:

$$p(y_n \mid \mathbf{x}_n, \boldsymbol\theta) = \mathcal{N}\big(y_n \mid \mathbf{x}_n^\top\boldsymbol\theta,\ \sigma^2\big)$$

Read the model this asserts: the label is the linear prediction plus Gaussian noise of known variance. Note what §8.2 did not have to say — page 803’s §8.2.5 quote, that empirical risk minimization never has to “specify the noise distribution for the labels”. Here you do.

Then, using independence:

$$\mathcal{L}(\boldsymbol\theta) = -\sum_{n=1}^{N}\log p(y_n \mid \mathbf{x}_n, \boldsymbol\theta)$$

Theta is on the right of the conditioning bar and is still the variable

The book flags the trap directly: “While it is tempting to interpret the fact that $\boldsymbol\theta$ is on the right of the conditioning in $p(y_n \mid \mathbf{x}_n, \boldsymbol\theta)$, and hence should be interpreted as observed and fixed, this interpretation is incorrect.”

Conditioning notation says “given” — but here it means “for this setting of”, not “having observed”. $\mathcal{L}$ is a function of $\boldsymbol\theta$ and we minimise over it.

Example 8.5: the derivation that collapses two frameworks into one

Substituting the Gaussian and expanding, step by step:

$$\mathcal{L}(\boldsymbol\theta) = -\sum_{n=1}^{N}\log \mathcal{N}\big(y_n \mid \mathbf{x}_n^\top\boldsymbol\theta,\ \sigma^2\big)$$ $$= -\sum_{n=1}^{N}\log\left[\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{(y_n - \mathbf{x}_n^\top\boldsymbol\theta)^2}{2\sigma^2}\right)\right]$$

The logarithm splits the product inside into a sum:

$$= -\sum_{n=1}^{N}\log\exp\left(-\frac{(y_n - \mathbf{x}_n^\top\boldsymbol\theta)^2}{2\sigma^2}\right) - \sum_{n=1}^{N}\log\frac{1}{\sqrt{2\pi\sigma^2}}$$

and $\log\exp$ cancels:

$$= \frac{1}{2\sigma^2}\sum_{n=1}^{N}\big(y_n - \mathbf{x}_n^\top\boldsymbol\theta\big)^2 - \sum_{n=1}^{N}\log\frac{1}{\sqrt{2\pi\sigma^2}}$$

The book’s conclusion: “As $\sigma$ is given, the second term in (8.18d) is constant, and minimizing $\mathcal{L}(\boldsymbol\theta)$ corresponds to solving the least-squares problem.”

What that means precisely

Compare the first term with Equation 8.8’s $R_{\text{emp}} = \frac{1}{N}\sum_n (y_n - \boldsymbol\theta^\top\mathbf{x}_n)^2$. They differ by the factor $\frac{N}{2\sigma^2}$. So

$$\mathcal{L}(\boldsymbol\theta) = \frac{N}{2\sigma^2}\,R_{\text{emp}}(\boldsymbol\theta) + c, \qquad c = -N\log\frac{1}{\sqrt{2\pi\sigma^2}}$$

A positive multiple plus a constant. Neither operation can move an $\arg\min$. Measured with $N = 25$ and $\sigma = 0.35$: the slope is $102.040816$, the constant is $-3.272090$, and the two minimisers agree to $1.8\times10^{-15}$.

So §8.2 and §8.3 are not two methods that happen to agree on this example. For a Gaussian likelihood with known variance they are the same objective in different notation, and the choice between them is a choice of vocabulary — until §8.3.2 adds a prior, where the vocabularies start to buy different things.

§8.3.2’s remark: what maximum likelihood guarantees

The book lists three properties (Lehmann and Casella, 1998; Efron and Hastie, 2016):

1. Asymptotic consistency. The MLE converges to the true value in the limit of infinitely many observations, plus a random error that is approximately normal.
2. The error’s variance decays as $1/N$.
3. And the caveat: “the size of the samples necessary to achieve these properties can be quite large.”

Then the warning that motivates the next page: “Especially, in the ‘small’ data regime, maximum likelihood estimation can lead to overfitting.”

Measured on a three-parameter model with known truth:

$N$	mean squared error	$N \times$ error
$10$	$0.15053755$	$1.505375$
$40$	$0.02453147$	$0.981259$
$160$	$0.00586250$	$0.938000$
$640$	$0.00145983$	$0.934290$
$2560$	$0.00037095$	$0.949635$
$10240$	$0.00009030$	$0.924683$

The right column is roughly constant from $N = 40$ onward, which is the $1/N$ claim. At $N = 10$ it is $1.51$ — noticeably above the asymptote, which is the “samples necessary can be quite large” caveat showing up at the small end.

Worked example by hand

Derive the MLE for the mean of a Gaussian, the smallest complete case.

Model: $y_n \sim \mathcal{N}(\theta, \sigma^2)$ with $\sigma$ known, and observations $y = (1.2,\ 2.1,\ 1.7)$.

Step 1: the negative log-likelihood.

$$\mathcal{L}(\theta) = -\sum_{n=1}^{3}\log\left[\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{(y_n - \theta)^2}{2\sigma^2}\right)\right] = \frac{1}{2\sigma^2}\sum_{n=1}^{3}(y_n - \theta)^2 + \text{const}$$

Step 2: differentiate and set to zero.

$$\frac{\mathrm{d}\mathcal{L}}{\mathrm{d}\theta} = -\frac{1}{\sigma^2}\sum_{n=1}^{3}(y_n - \theta) = 0 \quad\Longrightarrow\quad \sum_{n=1}^{3} y_n = 3\theta \quad\Longrightarrow\quad \theta = \frac{1}{3}\sum_{n=1}^{3}y_n$$

The maximum likelihood estimate of a Gaussian mean is the sample mean. With these numbers:

$$\hat\theta = \frac{1.2 + 2.1 + 1.7}{3} = \frac{5.0}{3} = 1.666667$$

Measured by a grid search over the likelihood: $1.666655$, differing only by the grid spacing.

Step 3: check it is a minimum. $\dfrac{\mathrm{d}^2\mathcal{L}}{\mathrm{d}\theta^2} = \dfrac{3}{\sigma^2} > 0$. Positive for every $\sigma$, so it is a minimum of $\mathcal{L}$ and hence a maximum of the likelihood.

Step 4: notice what $\sigma$ did. It appears in $\mathcal{L}$, it appears in the second derivative, and it cancels completely from the answer. That is the same cancellation as Equation 8.18’s: with $\sigma$ known and constant, it scales the objective without moving its minimiser. The MLE for the mean is the sample mean whatever the noise level is.

See it move

The two readings are much easier to feel than to read about:

sketch One formula, two readings p5.js Pause Restart

The top panel fixes theta and varies y, giving a density that integrates to one. The bottom panel fixes the data and varies theta, giving the likelihood — which does not. Drag theta and watch the marker move along the likelihood curve as the density slides.

From scratch

mle.py

import numpy as np
 
SIGMA = 0.35
 
def make(n, seed, noise=SIGMA):
    rng = np.random.default_rng(seed)
    x = np.sort(rng.uniform(-3, 3, n))
    return x, np.sin(1.4 * x) + 0.3 * x + noise * rng.standard_normal(n)
 
def design(x, deg):
    return np.vander(x / 3.0, deg + 1, increasing=True)
 
x, y = make(25, seed=3)
Phi = design(x, 3)
N = len(y)
 
def empirical_risk(th):
    """Equation 8.7."""
    return float(np.mean((y - Phi @ th) ** 2))
 
def nll(th):
    """Equation 8.18d, written exactly as the book leaves it."""
    r = y - Phi @ th
    return float(r @ r / (2 * SIGMA ** 2)
                 - N * np.log(1.0 / np.sqrt(2 * np.pi * SIGMA ** 2)))
 
# --- the two objectives have the same minimiser ---------------------------
th_ls = np.linalg.lstsq(Phi, y, rcond=None)[0]
 
# minimise the NLL from a different start by Newton's method
th = np.zeros(Phi.shape[1])
H = Phi.T @ Phi / SIGMA ** 2
for _ in range(50):
    g = -Phi.T @ (y - Phi @ th) / SIGMA ** 2
    th = th - np.linalg.solve(H, g)
 
print("least-squares theta :", np.round(th_ls, 6).tolist())
print("NLL-minimising theta:", np.round(th, 6).tolist())
print(f"max |difference|    : {np.abs(th - th_ls).max():.3e}")
 
# --- and they differ by an affine transformation --------------------------
const = -N * np.log(1.0 / np.sqrt(2 * np.pi * SIGMA ** 2))
slope = N / (2 * SIGMA ** 2)
print(f"\nEquation 8.18d says L = (1/(2 sigma^2)) sum r^2 + const")
print(f"  in terms of R_emp that is  L = {slope:.6f} * R_emp + "
      f"({const:.6f})")
print(f"\n{'theta tried':>36} {'R_emp':>12} {'L':>14} {'predicted L':>14}")
for scale in (0.0, 0.5, 1.0, 1.5):
    t = scale * th_ls
    pred = slope * empirical_risk(t) + const
    print(f"{str(np.round(t, 3).tolist()):>36} {empirical_risk(t):>12.6f} "
          f"{nll(t):>14.6f} {pred:>14.6f}")
print("the last two columns agree, so the NLL is an affine function of R_emp")
print("and a positive multiple plus a constant cannot move an argmin.")
 
# --- the likelihood is not a distribution over theta ---------------------
print("\nThe likelihood is NOT a distribution over theta:")
obs = np.array([1.2, 2.1, 1.7])
sig = 0.8
ths = np.linspace(-10, 12, 400001)
lik = np.ones_like(ths)
for o in obs:
    lik *= np.exp(-0.5 * ((o - ths) / sig) ** 2) / (sig * np.sqrt(2 * np.pi))
print(f"  three observations {obs.tolist()}, sigma = {sig}")
print(f"  area under the likelihood in theta: {np.trapezoid(lik, ths):.6f}")
print(f"  argmax at theta = {ths[int(np.argmax(lik))]:.6f}, "
      f"sample mean = {obs.mean():.6f}")
ys = np.linspace(-12, 14, 400001)
dens = np.exp(-0.5 * ((ys - 2.0) / sig) ** 2) / (sig * np.sqrt(2 * np.pi))
print(f"  area under p(y | theta=2) in y:     {np.trapezoid(dens, ys):.6f}")
print("  one integrates to 1, the other does not. Only one is a density.")
 
# --- Section 8.3.2's remark: the error variance decays as 1/N -----------
print("\nThe MLE error variance decays as 1/N:")
true_theta = np.array([0.4, 1.2, -0.7])
rng = np.random.default_rng(0)
print(f"{'N':>8} {'mean sq error':>16} {'N x error':>12}")
for n in (10, 40, 160, 640, 2560, 10240):
    trials = 2000
    errs = np.empty(trials)
    for t in range(trials):
        xs = rng.uniform(-2, 2, n)
        A = np.vander(xs, 3, increasing=True)
        ys_ = A @ true_theta + 0.5 * rng.standard_normal(n)
        thn = np.linalg.lstsq(A, ys_, rcond=None)[0]
        errs[t] = np.sum((thn - true_theta) ** 2)
    m = errs.mean()
    print(f"{n:>8} {m:>16.8f} {n * m:>12.6f}")
print("N times the error is roughly constant, which is the 1/N claim.")

text

least-squares theta : [-0.116506, 3.633144, 0.492278, -4.086057]
NLL-minimising theta: [-0.116506, 3.633144, 0.492278, -4.086057]
max |difference|    : 1.776e-15
 
Equation 8.18d says L = (1/(2 sigma^2)) sum r^2 + const
  in terms of R_emp that is  L = 102.040816 * R_emp + (-3.272090)
 
                         theta tried        R_emp              L    predicted L
              [-0.0, 0.0, 0.0, -0.0]     0.998583      98.624110      98.624110
      [-0.058, 1.817, 0.246, -2.043]     0.359425      33.403965      33.403965
      [-0.117, 3.633, 0.492, -4.086]     0.146373      11.663917      11.663917
       [-0.175, 5.45, 0.738, -6.129]     0.359425      33.403965      33.403965
the last two columns agree, so the NLL is an affine function of R_emp
and a positive multiple plus a constant cannot move an argmin.
 
The likelihood is NOT a distribution over theta:
  three observations [1.2, 2.1, 1.7], sigma = 0.8
  area under the likelihood in theta: 0.104496
  argmax at theta = 1.666655, sample mean = 1.666667
  area under p(y | theta=2) in y:     1.000000
  one integrates to 1, the other does not. Only one is a density.
 
The MLE error variance decays as 1/N:
       N    mean sq error    N x error
      10       0.15053755     1.505375
      40       0.02453147     0.981259
     160       0.00586250     0.938000
     640       0.00145983     0.934290
    2560       0.00037095     0.949635
   10240       0.00009030     0.924683
N times the error is roughly constant, which is the 1/N claim.

On real data

figure One formula and two different objects matplotlib

Fix theta and vary y and you have a distribution that integrates to one. Fix y and vary theta and you have the likelihood, whose area here is 0.104496. The three squares on the right are the same three curves from the left, read as three points on one function.

figure Equation 8.18: Gaussian maximum likelihood is least squares matplotlib

The two objectives differ by a positive multiple and a constant, and neither operation can move a minimiser. Plotting one against the other gives a straight line to within floating-point error.

figure Section 8.3.2's remark, checked matplotlib

N times the mean squared error is roughly constant from N equals 40 onward, confirming the one-over-N decay. The error distribution is close to normal. But with only twelve points the same estimator overfits badly, which is the remark's closing warning.

Reading the plot

The first figure is the distinction the whole section turns on. Both panels are built from the same formula and the same three observations. On the left, $\boldsymbol\theta$ is fixed at three different values and $y$ runs along the axis: three bell curves, each integrating to $1$, each a legitimate probability density over data. The green lines are the observations, and you can see each curve assigning them different densities.

On the right, the observations are fixed and $\boldsymbol\theta$ runs along the axis. The three coloured squares are the same three settings — the product of the three green dots’ heights, for each curve. Sweeping $\boldsymbol\theta$ traces out the likelihood.

Now the part that matters: the annotation says the area under the right-hand curve is $0.104496$. It is not $1$, it was never going to be $1$, and nothing in the construction normalises it. The likelihood is a function of $\boldsymbol\theta$, not a distribution over $\boldsymbol\theta$. Phrases like “the most likely parameter” are convenient and slightly wrong; the object that is a distribution over $\boldsymbol\theta$ is the posterior, and getting it requires the prior of §8.3.2.

The second figure turns “analogous” into “identical”. The left panel slices both objectives along one coordinate. The blue curve ($R_{\text{emp}}$) and the green curve ($\mathcal{L}$) are on different axes with different scales — but their minima are at the same place, to six decimals.

The right panel is why. Plot $\mathcal{L}$ directly against $R_{\text{emp}}$ and you get a straight line: slope $N/(2\sigma^2) = 102.040816$, intercept $-3.272090$, with a maximum deviation at the $10^{-14}$ level. An affine map with positive slope is monotone, so it preserves ordering exactly — every $\boldsymbol\theta$ that one objective prefers, the other prefers too.

This is stronger than an analogy and it is worth being clear about the scope. It holds for a Gaussian likelihood with known, constant variance. Change the noise model and the correspondence breaks: a Laplace likelihood gives absolute loss, not squared. Let $\sigma$ depend on $\mathbf{x}$ and the sum becomes weighted least squares. The general statement is that choosing a noise distribution is choosing a loss function — and §8.2’s freedom not to specify one is exactly the freedom §8.3 gives up.

The third figure checks the three claims and the caveat. The left panel plots the mean squared error against $N$ on log-log axes with a pure $1/N$ reference. The measured products $N \times \text{error}$ run $1.505$, $0.981$, $0.938$, $0.934$, $0.950$, $0.925$ — settled by $N = 40$ and constant thereafter. The one visible outlier is $N = 10$, sitting $60\%$ above the asymptote, which is the book’s “the size of the samples necessary to achieve these properties can be quite large” made visible at the small end.

The middle panel checks “approximately normal”: standardised errors against a standard normal, with measured skew and excess kurtosis both near zero. For a linear-Gaussian model this is exact rather than asymptotic, which is why the agreement is so clean — a nonlinear model would show visible departure at small $N$.

The right panel is the warning, and it is the bridge to page 805. With $N = 12$ the same estimator, with no bug and no misconfiguration, produces an expected risk that climbs steeply with model complexity. The MLE has no mechanism to prevent this: it maximises fit to the data it has, full stop. §8.2.3 added a penalty to fix that; §8.3.2 adds a prior, and page 805 measures that they are the same repair.

From the book

This page covers §8.3 and §8.3.1 of Mathematics for Machine Learning by Deisenroth, Faisal and Ong (draft of 2019-12-11), plus the remark on MLE properties that opens §8.3.2. The framing that the likelihood is analogous to a loss function and the prior to regularisation is §8.3’s. Equation 8.14 defines the negative log-likelihood; the two readings of $p(\mathbf{x} \mid \boldsymbol\theta)$ are given in the text, as is the note that the subscript is usually dropped. Equations 8.15 to 8.17 set up the supervised case, and the warning that $\boldsymbol\theta$ on the right of the conditioning bar is still the variable is the book’s. The remark that the negative sign is “a historical artifact” due to the minimisation convention is verbatim. Example 8.5 is Equations 8.18a through 8.18d and the conclusion that minimising $\mathcal{L}$ solves the least-squares problem; Figure 8.5 shows the MLE fit with its prediction at $x = 60$. The three MLE properties — asymptotic consistency, approximately normal error, variance decaying as $1/N$ — cite Lehmann and Casella (1998) and Efron and Hastie (2016), and the closing warning about the small-data regime is the book’s. The measured $1.8\times10^{-15}$ agreement, the affine relation $\mathcal{L} = 102.040816\,R_{\text{emp}} - 3.272090$, the likelihood’s area of $0.104496$, and the $N \times$ error table are this module’s additions.

Pitfalls

Treating the likelihood as a distribution over the parameters

Measured: the likelihood in the worked example integrates to $0.104496$ over $\theta$, while the density it was built from integrates to $1.000000$ over $y$. Nothing normalises the likelihood in $\boldsymbol\theta$ and nothing requires it to be finite. “The most likely $\boldsymbol\theta$” is a figure of speech; the object that genuinely is a distribution over $\boldsymbol\theta$ is the posterior, and it needs a prior.

Reading theta on the right of the bar as observed

The book warns about this explicitly. In $p(y_n \mid \mathbf{x}_n, \boldsymbol\theta)$ the conditioning notation means “for this setting of”, not “having observed”. $\mathcal{L}$ is a function of $\boldsymbol\theta$ and you minimise over it.

Assuming maximum likelihood always means least squares

It means least squares for a Gaussian likelihood with known, constant variance — that is what Equation 8.18’s cancellation needs. A Laplace likelihood gives absolute loss. An input-dependent $\sigma(\mathbf{x})$ gives weighted least squares. Choosing a noise model is choosing a loss function, and the correspondence is specific rather than general.

Expecting the asymptotic properties at small N

Consistency, normality and $1/N$ variance are all asymptotic, and the book adds that the samples needed “can be quite large”. Measured, $N \times \text{error}$ was $1.505$ at $N = 10$ against an asymptote near $0.93$ — $60\%$ high. And with $N = 12$ the MLE overfits badly at any nontrivial model size. The guarantees do not arrive gradually enough to rely on in the small-data regime.

Dropping the log and optimising the likelihood directly

The log is not cosmetic. Independence makes the likelihood a product over examples, and §0.3 measured a product of $162$ probabilities of $0.01$ reaching exactly 0.0 in float64 — at which point the gradient is undefined too. Every likelihood in Chapters 9 to 12 is logged before it is optimised.

Forgetting that sigma cancels

In the worked example $\sigma$ appears in $\mathcal{L}$ and in its second derivative, and vanishes from the answer: the MLE of a Gaussian mean is the sample mean regardless of the noise level. That is the same cancellation as Equation 8.18’s. It stops being true the moment $\sigma$ is itself estimated, or varies with the input.

Compare

	empirical risk minimization, §8.2	maximum likelihood, §8.3.1
the object minimised	$R_{\text{emp}}$, Eq 8.6	$\mathcal{L}$, Eq 8.14
what you must specify	a loss function	a noise distribution
the Gaussian case	squared loss	the same objective, scaled
measured relation	—	$\mathcal{L} = 102.040816\,R_{\text{emp}} - 3.272090$
minimisers agree to	—	$1.8\times10^{-15}$
regularisation appears as	a penalty term, §8.2.3	a prior, §8.3.2
output	a point estimate	a point estimate
can express predictive uncertainty	no	yes, via the noise model

noise model	the loss it implies
Gaussian, constant $\sigma$	squared loss
Gaussian, $\sigma(\mathbf{x})$	weighted squared loss
Laplace	absolute loss
Bernoulli	cross-entropy / logistic loss

pch.quizTag Do you know what the likelihood is and is not?

1. Why does the likelihood in the worked example integrate to 0.104496 rather than 1?

   A Because of numerical error in the integration B Because it is a function of theta and was never normalised in theta — it is not a density over parameters C Because three observations are too few D Because sigma was set incorrectly
   pch.quizShowAnswer

   B — Because it is a function of theta and was never normalised in theta — it is not a density over parameters — The density p(y | theta) integrates to exactly 1.000000 over y, because that is what a density does. The likelihood is the same formula read along the other axis, and nothing normalises it there. Turning it into a distribution over theta requires a prior, which is Section 8.3.2.

2. Equation 8.18 shows the Gaussian negative log-likelihood equals (N / 2 sigma squared) times the empirical risk, plus a constant. What follows?

   A The two objectives give approximately similar answers B They have identical minimisers, because a positive multiple plus a constant cannot move an argmin — measured agreement 1.8e-15 C The empirical risk is a special case of the likelihood D The constant must be subtracted before optimising
   pch.quizShowAnswer

   B — They have identical minimisers, because a positive multiple plus a constant cannot move an argmin — measured agreement 1.8e-15 — An affine map with positive slope is strictly monotone, so it preserves ordering exactly: every theta one objective prefers, the other prefers too. Section 8.2 and Section 8.3 are the same objective in different vocabulary — for this noise model.

3. Does maximum likelihood always reduce to least squares?

   A Yes, that is what Equation 8.18 proves B No — only for a Gaussian likelihood with known constant variance. A Laplace likelihood gives absolute loss, and an input-dependent sigma gives weighted least squares C Only when the model is linear D Only when N is large
   pch.quizShowAnswer

   B — No — only for a Gaussian likelihood with known constant variance. A Laplace likelihood gives absolute loss, and an input-dependent sigma gives weighted least squares — The cancellation in Equation 8.18 depends on the specific form of the Gaussian and on sigma being constant. The general statement is that choosing a noise distribution IS choosing a loss function — which is exactly the specification Section 8.2 was free to skip.

4. Why is the negative sign in Equation 8.14 there?

   A To make the objective convex B It is a historical artifact: we want to maximise likelihood, but optimisation literature studies minimisation C To make the log well defined D To cancel the sign of the exponent in the Gaussian
   pch.quizShowAnswer

   B — It is a historical artifact: we want to maximise likelihood, but optimisation literature studies minimisation — The book says exactly this. Negation reverses the ordering once, so argmax of p equals argmin of L with no approximation. The logarithm, by contrast, IS doing work — it turns a product that underflows into a sum that does not.

5. N times the MLE's mean squared error measured 1.505 at N = 10 and settled near 0.93 from N = 40 onward. What does that show?

   A The estimator is biased B The 1 over N decay is confirmed asymptotically, and N = 10 is visibly outside the regime where the asymptotic properties hold C The model was misspecified D The variance grows with N
   pch.quizShowAnswer

   B — The 1 over N decay is confirmed asymptotically, and N = 10 is visibly outside the regime where the asymptotic properties hold — A constant product means error proportional to one over N, which is the book's claim. The N = 10 value sitting 60 percent above the asymptote is the book's own caveat — that the sample sizes needed can be quite large — showing up at the small end, and it is why Section 8.3.2 exists.

🧪 Try It Yourself

Exercise 1 – The MLE of a Gaussian mean

pch.exerciseTag pch.exerciseTitle

Exercise 2 – The likelihood is not a density over theta

pch.exerciseTag pch.exerciseTitle

Exercise 3 – Equation 8.18, the affine relation

pch.exerciseTag pch.exerciseTitle

Exercise 4 – Both routes reach the same theta

pch.exerciseTag pch.exerciseTitle

Exercise 5 – The 1/N decay, and where it fails

pch.exerciseTag pch.exerciseTitle

Recall card

• Section 8.3’s own mapping: the likelihood is analogous to a loss function, and a prior is analogous to regularisation. Measured, “analogous” understates it — for a Gaussian the two are affine transforms of each other.
• Equation 8.14 is minus the log of the likelihood. The log turns an underflowing product into a sum; the negative sign is a historical artifact of the minimisation convention. Neither moves the argmin.
• p(x given theta) is one formula and two objects. Fix theta and vary x: a distribution over data, integrating to one. Fix x and vary theta: the likelihood, which does not.
• Measured: the likelihood integrated to 0.104496 over theta while the density integrated to 1.000000 over y. “The most likely theta” is a figure of speech; the object that IS a distribution over theta is the posterior.
• Theta on the right of the conditioning bar is still the variable. The book warns about this explicitly — the bar means “for this setting of”, not “having observed”.
• Equations 8.15 to 8.17: a Gaussian conditional on the linear prediction, then independence turns the product into a sum of logs.
• Example 8.5’s punchline: the Gaussian NLL is (N over 2 sigma squared) times the empirical risk plus a constant. Measured with N = 25 and sigma = 0.35, that is 102.040816 times R_emp minus 3.272090, and the two minimisers agree to 1.8e-15.
• A positive multiple plus a constant cannot move an argmin, so Section 8.2 and Section 8.3 are the same objective in different vocabulary.
• But only for THIS noise model. Laplace gives absolute loss; an input-dependent sigma gives weighted least squares. Choosing a noise distribution IS choosing a loss function — the specification Section 8.2 was free to skip.
• Sigma cancels from the answer. In the worked example it appears in the NLL and in its second derivative and vanishes from the estimate: the MLE of a Gaussian mean is the sample mean at any noise level.
• Three asymptotic properties: consistency, approximately normal error, and error variance decaying as one over N. Measured, N times the error settles near 0.93 from N = 40 onward.
• And one caveat that bites. At N = 10 the product was 1.505, 60 percent above the asymptote, and with N = 12 the MLE overfits badly. The book’s closing line is that in the small-data regime maximum likelihood can lead to overfitting — which is what the next page fixes.

Next: add a prior, and discover it was the penalty term all along. MAP Estimation and Model Fitting