Chapter 6 Formula Sheet

Reference, not teaching. Each entry names the book’s number, states the result, and gives the condition it needs. Read the concept pages first; come back here when you are working.

The one-line summary

Two rules generate the chapter, and one family makes them computable:

$$\underbrace{\text{sum rule}}_{\text{Eq 6.20}} \;+\; \underbrace{\text{product rule}}_{\text{Eq 6.22}} \;\Longrightarrow\; \underbrace{\text{Bayes}}_{\text{Eq 6.23}} \;\xrightarrow[\text{Eq 6.107}]{\text{exponential family}}\; \underbrace{\text{closed-form inference}}_{\text{conjugacy}}$$

Everything else is either a named distribution, a summary statistic, or a rule for transforming one distribution into another.

The three objects, and the fourth

Object	Symbol	Lives in	§6.1
sample space	$\Omega$	—	the set of outcomes
event space	$\mathcal{A}$	subsets of $\Omega$	the askable questions; the power set for discrete
probability	$P$	$\mathcal{A}\to[0,1]$	a measure, $P(\Omega)=1$
target space	$\mathcal{T}$	—	the quantity of interest
random variable	$X$	$\Omega\to\mathcal{T}$	a function, not a variable
law / distribution	$P_X$	$\mathcal{T}\to[0,1]$	$P\circ X^{-1}$

$$P_X(S) = P(X\in S) = P\bigl(X^{-1}(S)\bigr) = P\bigl(\{\omega\in\Omega : X(\omega)\in S\}\bigr) \tag{6.8}$$

The pre-image is the whole point

$X$ may be non-injective, so several outcomes can share a state. In Example 6.1 two outcomes map to “one dollar coin”, so $P(X=1) = 0.21+0.21 = 0.42$, not $0.21$. Any formula that does not sum over the pre-image gets half the right answer, and the pmf then fails to sum to $1$ — which is your check.

§6.2 Discrete and continuous

#	Result	Condition
Eq 6.9	joint $P(X=x_i,Y=y_j) = n_{ij}/N$	discrete
Eq 6.10, 6.11	marginals $c_i/N$ and $r_j/N$	discrete
Eq 6.12	probabilities sum to 1	discrete, finite
Eq 6.13	$P(Y=y_j\mid X=x_i) = n_{ij}/c_i$ — normalises columns	—
Eq 6.14	$P(X=x_i\mid Y=y_j) = n_{ij}/r_j$ — normalises rows	—
Def 6.1	pdf: $f(\mathbf{x})\geqslant0$ and $\int f = 1$ (Eq 6.15)	no upper bound on $f$
Eq 6.16	$P(a\leqslant X\leqslant b) = \int_a^b f$	continuous
Def 6.2	cdf $F_X(\mathbf{x}) = P(X_1\leqslant x_1,\ldots)$ (Eq 6.17, 6.18)	some cdfs have no pdf

Table 6.1, the nomenclature:

Type	“point probability”	“interval probability”
discrete	$P(X=x)$, a pmf	not applicable
continuous	$p(x)$, a pdf	$P(X\leqslant x)$, a cdf

A density is not a probability

Definition 6.1 constrains the area, not the height. Measured: the uniform on $[0.9,1.6]$ has density $1.4286$; a Gaussian with $\sigma=0.01$ peaks at $39.89$; a uniform of width $10^{-8}$ reaches $10^{8}$ — integral exactly $1$ in every case. And $P(X=x) = 0$ exactly for continuous $X$: a million uniform samples gave zero exact hits.

§6.3 The two rules and Bayes

$$p(\mathbf{x}) = \sum_{\mathbf{y}} p(\mathbf{x},\mathbf{y}) \quad\text{or}\quad \int p(\mathbf{x},\mathbf{y})\,\mathrm{d}\mathbf{y} \tag{6.20}$$ $$p(x_i) = \int p(x_1,\ldots,x_D)\,\mathrm{d}\mathbf{x}_{\setminus i} \tag{6.21}$$ $$p(\mathbf{x},\mathbf{y}) = p(\mathbf{y}\mid\mathbf{x})p(\mathbf{x}) = p(\mathbf{x}\mid\mathbf{y})p(\mathbf{y}) \tag{6.22}$$ $$\underbrace{p(\mathbf{x}\mid\mathbf{y})}_{\text{posterior}} = \frac{\overbrace{p(\mathbf{y}\mid\mathbf{x})}^{\text{likelihood}}\;\overbrace{p(\mathbf{x})}^{\text{prior}}}{\underbrace{p(\mathbf{y})}_{\text{evidence}}} \tag{6.23}$$ $$p(\mathbf{y}) := \int p(\mathbf{y}\mid\mathbf{x})p(\mathbf{x})\,\mathrm{d}\mathbf{x} = \mathbb{E}_X\bigl[p(\mathbf{y}\mid\mathbf{x})\bigr] \tag{6.27}$$

Quantity	Normalises over	Notes
prior $p(x)$	$x$	must be nonzero on every plausible state
likelihood $p(y\mid x)$	$y$, not $x$	never “the likelihood of $y$”
evidence $p(y)$	— (a number)	expected likelihood under the prior
posterior $p(x\mid y)$	$x$	the quantity of interest

Three measured traps

The sum rule is the expensive one. $D$ binary variables give $2^D$ joint entries — $1.27\times10^{30}$ at $D=100$ — with no known exact polynomial-time algorithm.

The base-rate fallacy. Sensitivity and specificity both $0.99$ at prevalence $0.001$ gives $P(\text{disease}\mid+) = 0.0902$, not $0.99$ — $10.98\times$ overstated. In $100\,000$ people: $99$ true positives against $999$ false.

A zero prior is permanent. Measured at exactly 0.0000000000 after $400$ confirming observations, because Equation 6.23 multiplies. And the failure is silent — the remaining states still normalise.

§6.4 Summary statistics

#	Result	Shape / condition
Def 6.3	$\mathbb{E}_X[g(x)] = \int g(x)p(x)\,\mathrm{d}x$ or $\sum g(x)p(x)$	Eq 6.28–6.30
Eq 6.34	$\mathbb{E}[ag+bh] = a\mathbb{E}[g]+b\mathbb{E}[h]$	no independence needed
Def 6.4	mean, elementwise	Eq 6.31, 6.32
Def 6.5	$\operatorname{Cov}[x,y] = \mathbb{E}[(x-\mathbb{E}x)(y-\mathbb{E}y)]$	Eq 6.35
Eq 6.36	$= \mathbb{E}[xy]-\mathbb{E}[x]\mathbb{E}[y]$	—
Def 6.6	$\operatorname{Cov}[\mathbf{x},\mathbf{y}] = \mathbb{E}[\mathbf{x}\mathbf{y}^\top]-\mathbb{E}[\mathbf{x}]\mathbb{E}[\mathbf{y}]^\top$	$D\times E$, Eq 6.37
Def 6.7	covariance matrix, Eq 6.38c	symmetric, positive semidefinite
Def 6.8	$\operatorname{corr}[x,y] = \operatorname{Cov}/\sqrt{\mathbb{V}[x]\mathbb{V}[y]}$	in $[-1,1]$, Eq 6.40
Def 6.9	empirical mean and covariance, Eq 6.41, 6.42	the book’s is biased ($1/N$)
Eq 6.43	$\mathbb{V}[x] = \mathbb{E}[(x-\mu)^2]$	two passes, numerically safe
Eq 6.44	$= \mathbb{E}[x^2]-\mathbb{E}[x]^2$	one pass, unsafe
Eq 6.45	pairwise sum over $N^2$ terms $=$ twice Eq 6.44	—
Eq 6.46–6.49	sums; variances need the cross terms	—
Eq 6.50, 6.51	$\mathbb{E}[\mathbf{A}\mathbf{x}+\mathbf{b}] = \mathbf{A}\boldsymbol{\mu}+\mathbf{b}$, $\mathbb{V} = \mathbf{A}\boldsymbol{\Sigma}\mathbf{A}^\top$	takes the output’s shape
Eq 6.52	$\operatorname{Cov}[\mathbf{x},\mathbf{y}] = \boldsymbol{\Sigma}\mathbf{A}^\top$	—
Def 6.10	independence: $p(\mathbf{x},\mathbf{y}) = p(\mathbf{x})p(\mathbf{y})$	Eq 6.53
Def 6.11	conditional independence, Eq 6.55	must hold for every $\mathbf{z}$
Eq 6.57	equivalently $p(x\mid y,z) = p(x\mid z)$	—
Eq 6.59–6.61	covariance as an inner product; sd is a length, correlation a cosine	zero mean

Equation 6.44 destroys your answer

Variance is shift-invariant, so adding a constant must change nothing. Measured on eight numbers whose variance is $4$:

offset	Eq 6.43	Eq 6.44
$0$	$4.0000000000$	$4.0000000000$
$10^8$	$4.0000000000$	$4.0000000000$
$10^9$	$4.0000000000$	$0.0000000000$

Unix timestamps sit at $1.7\times10^9$. Use Equation 6.43 or Welford.

Zero covariance is not independence

Example 6.5: $y = x^2$ with symmetric $x$ gives $\operatorname{Cov} = 0$ while $y$ is a deterministic function of $x$ — total variation distance from independence $0.406$, and conditioning on $\lvert x\rvert<0.5$ cuts $\mathbb{V}[y]$ from $1.993$ to $0.005$. Covariance measures linear dependence only.

§6.5 The Gaussian

$$p(x\mid\mu,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) \tag{6.62}$$ $$p(\mathbf{x}\mid\boldsymbol{\mu},\boldsymbol{\Sigma}) = (2\pi)^{-\frac D2}\lvert\boldsymbol{\Sigma}\rvert^{-\frac12}\exp\!\left(-\tfrac12(\mathbf{x}-\boldsymbol{\mu})^\top\boldsymbol{\Sigma}^{-1}(\mathbf{x}-\boldsymbol{\mu})\right) \tag{6.63}$$

Operation	Result	#
marginalise	$\mathcal{N}(\boldsymbol{\mu}_x,\boldsymbol{\Sigma}_{xx})$ — delete rows and columns	6.68
condition	$\boldsymbol{\mu}_x+\boldsymbol{\Sigma}_{xy}\boldsymbol{\Sigma}_{yy}^{-1}(\mathbf{y}-\boldsymbol{\mu}_y)$	6.66
	$\boldsymbol{\Sigma}_{xx}-\boldsymbol{\Sigma}_{xy}\boldsymbol{\Sigma}_{yy}^{-1}\boldsymbol{\Sigma}_{yx}$	6.67
multiply	$\mathbf{C} = (\mathbf{A}^{-1}+\mathbf{B}^{-1})^{-1}$ — precisions add	6.74
	$\mathbf{c} = \mathbf{C}(\mathbf{A}^{-1}\mathbf{a}+\mathbf{B}^{-1}\mathbf{b})$	6.75
	$c = \mathcal{N}(\mathbf{a}\mid\mathbf{b},\mathbf{A}+\mathbf{B})$	6.76, 6.77
add independent	$\mathcal{N}(\boldsymbol{\mu}_x+\boldsymbol{\mu}_y,\ \boldsymbol{\Sigma}_x+\boldsymbol{\Sigma}_y)$	6.78
weighted sum	$\mathcal{N}(a\boldsymbol{\mu}_x+b\boldsymbol{\mu}_y,\ a^2\boldsymbol{\Sigma}_x+b^2\boldsymbol{\Sigma}_y)$	6.79
affine map	$\mathcal{N}(\mathbf{A}\boldsymbol{\mu},\ \mathbf{A}\boldsymbol{\Sigma}\mathbf{A}^\top)$ — may be singular	6.88
reverse map	mean $(\mathbf{A}^\top\mathbf{A})^{-1}\mathbf{A}^\top\mathbf{y}$ — the pseudo-inverse	6.91
mix densities	not Gaussian	Thm 6.12

Theorem 6.12, the mixture:

$$\mathbb{E}[x] = \alpha\mu_1+(1-\alpha)\mu_2 \tag{6.81}$$ $$\mathbb{V}[x] = \underbrace{\alpha\sigma_1^2+(1-\alpha)\sigma_2^2}_{\text{within}} + \underbrace{\alpha\mu_1^2+(1-\alpha)\mu_2^2 - \bigl[\alpha\mu_1+(1-\alpha)\mu_2\bigr]^2}_{\text{between}} \tag{6.82}$$

an instance of the law of total variance, $\mathbb{V}[x] = \mathbb{E}_Y[\mathbb{V}[x\mid y]] + \mathbb{V}_Y[\mathbb{E}[x\mid y]]$.

§6.5.4 sampling. If $\mathbf{x}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ then $\mathbf{y} = \mathbf{A}\mathbf{x}+\boldsymbol{\mu}$ has covariance $\mathbf{A}\mathbf{A}^\top$. Choose $\mathbf{A}$ as the Cholesky factor: it exists because covariances are symmetric positive definite, and it is triangular.

Conditioning can only shrink; a mixture is not a Gaussian

Equation 6.67 subtracts a positive-semidefinite term, so the variance never rises — measured eigenvalues $[0,\ 0.186,\ 1.847]$ on a 5-D example, and if $\boldsymbol{\Sigma}_{xy} = \mathbf{0}$ the observation teaches nothing.

And Theorem 6.12’s variance is not the average of the variances: measured within $1.0$, between $8.64$, total $9.64$ — the naive answer is $9.64\times$ too small. A moment-matched Gaussian has excess kurtosis $+0.003$ against the mixture’s $-1.475$, and one mode against two.

§6.6 Conjugacy and the exponential family

Distribution	Density	Mean	Variance
Bernoulli, Eq 6.92	$\mu^x(1-\mu)^{1-x}$	$\mu$	$\mu(1-\mu)$
Binomial, Eq 6.95	$\binom Nm\mu^m(1-\mu)^{N-m}$	$N\mu$	$N\mu(1-\mu)$
Beta, Eq 6.98	$\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}\mu^{\alpha-1}(1-\mu)^{\beta-1}$	$\frac{\alpha}{\alpha+\beta}$	Eq 6.99

The Beta’s four regimes: $\alpha=\beta=1$ uniform; $\alpha,\beta<1$ bimodal with endpoint spikes; $\alpha,\beta>1$ unimodal; $\alpha=\beta>1$ symmetric with mode $\tfrac12$.

Definition 6.13. A prior is conjugate for a likelihood if the posterior stays in the prior’s family. Table 6.2:

Likelihood	Conjugate prior
Bernoulli, Binomial	Beta
Gaussian (univariate)	Gaussian / inverse Gamma
Gaussian (multivariate)	Gaussian / inverse Wishart
Multinomial	Dirichlet

$$\text{Beta}(\alpha,\beta) + h \text{ heads in } N \Longrightarrow \operatorname{Beta}(h+\alpha,\ N-h+\beta) \tag{6.104d}$$

Theorem 6.14 (Fisher–Neyman). $\phi(\mathbf{x})$ is sufficient for $\boldsymbol{\theta}$ iff

$$p(\mathbf{x}\mid\boldsymbol{\theta}) = h(\mathbf{x})\,g_{\boldsymbol{\theta}}\bigl(\phi(\mathbf{x})\bigr) \tag{6.106}$$

The exponential family:

$$p(\mathbf{x}\mid\boldsymbol{\theta}) = h(\mathbf{x})\exp\bigl(\langle\boldsymbol{\theta},\phi(\mathbf{x})\rangle - A(\boldsymbol{\theta})\bigr) \tag{6.107}$$

Piece	Name	Role
$\phi(\mathbf{x})$	sufficient statistics	the only route from data to parameters
$\boldsymbol{\theta}$	natural parameters	unconstrained
$A(\boldsymbol{\theta})$	log-partition function	normalises; $\partial A/\partial\boldsymbol{\theta} = \mathbb{E}[\phi]$
$h(\mathbf{x})$	base measure	parameter-free

Distribution	$\phi$	$\theta$	$A(\theta)$
Gaussian, Ex 6.13	$(x, x^2)$	$\bigl(\mu/\sigma^2,\ -1/(2\sigma^2)\bigr)$	—
Bernoulli, Ex 6.14	$x$	$\log\frac{\mu}{1-\mu}$	$\log(1+e^\theta)$
Binomial	$m$	$\log\frac{\mu}{1-\mu}$	$N\log(1+e^\theta)$
Beta	$(\log\mu,\ \log(1-\mu))$	$(\alpha-1,\ \beta-1)$	$\log B(\alpha,\beta)$

$$\mu = \frac{1}{1+\exp(-\theta)} \tag{6.118}$$

the sigmoid — which is why logistic regression is a Bernoulli likelihood in natural parameters. And every member has a conjugate prior by formula:

$$p(\boldsymbol{\theta}\mid\boldsymbol{\gamma}) = h_c(\boldsymbol{\theta})\exp\!\left(\left\langle\begin{bmatrix}\gamma_1\\\gamma_2\end{bmatrix},\begin{bmatrix}\boldsymbol{\theta}\\-A(\boldsymbol{\theta})\end{bmatrix}\right\rangle - A_c(\boldsymbol{\gamma})\right) \tag{6.120}$$

Why conjugacy works, in one line

Multiplying exponential-family densities adds natural parameters. So a Beta prior times a Binomial likelihood has $\boldsymbol{\theta} = (\alpha-1+m,\ \beta-1+N-m)$, which is $\operatorname{Beta}(\alpha+m,\ \beta+N-m)$. Conjugacy is structural, not lucky — and Pitman, Darmois and Koopman proved (1935–36) that exponential families are the only ones with finite-dimensional sufficient statistics under repeated sampling.

§6.7 Change of variables

Case	Formula	#
discrete	$P(Y=y) = P\bigl(X=U^{-1}(y)\bigr)$ — no Jacobian	6.125
continuous, univariate	$f(y) = f_x\bigl(U^{-1}(y)\bigr)\left\lvert\frac{\mathrm{d}}{\mathrm{d}y}U^{-1}(y)\right\rvert$	6.143
continuous, multivariate	$f(\mathbf{y}) = f_x\bigl(U^{-1}(\mathbf{y})\bigr)\left\lvert\det\frac{\partial}{\partial\mathbf{y}}U^{-1}(\mathbf{y})\right\rvert$	Thm 6.16, 6.144

Theorem 6.15 (probability integral transform). For continuous $X$ with a strictly monotonic cdf, $Y := F_X(X)$ is uniform. Run backwards, that is inverse-transform sampling.

Two measured traps

The Jacobian is not optional. Dropping $\lvert1/y\rvert$ for $Y=\exp(X)$ gives a function integrating to $1.6470952$, and even after renormalising the worst pointwise error is $0.5275$ against $0.0047$. It is a shape error, so no constant repairs it.

The mode is not reparameterisation-invariant. $X\sim\mathcal{N}(0,1)$ has mode $0$; $Y=\exp(X)$ has mode $\exp(-1) = 0.3679$, not $\exp(0)=1$. Quantiles — and hence the median — are invariant. Only affine maps leave the mode alone, because their Jacobian is constant.

Numbers worth remembering

Quantity	Value	Where
Example 6.1’s pmf	$0.49,\ 0.42,\ 0.09$	§6.1
frequentist error rate	$\propto 1/\sqrt N$	§6.1
power set size	$2^{\lvert\Omega\rvert}$; $1.15\times10^{77}$ at $256$	§6.1
uniform on $[0.9,1.6]$ density	$1.4286 > 1$	§6.2
Gaussian peak at $\sigma=0.01$	$39.89$	§6.2
joint entries, $D$ binary	$2^D$; $1.27\times10^{30}$ at $D=100$	§6.3
base-rate overstatement	$10.98\times$ at prevalence $0.001$	§6.3
Eq 6.44 failure offset	$10^9$ returns $0$ for a variance of $4$	§6.4
biased covariance at $N=2$	half the true variance	§6.4
Example 6.6 conditional	$\mathcal{N}(0.6,\ 0.1)$ from $\mathcal{N}(0,\ 0.3)$	§6.5
mixture variance decomposition	within $1.0$, between $8.64$, total $9.64$	§6.5
mixture excess kurtosis	$-1.4746$ against a Gaussian’s $0$	§6.5
conjugate posterior size	$2$ numbers after $100\,000$ observations	§6.6
grid posterior size	$200^D$; $10^{23}$ at $D=10$	§6.6
sufficiency gap	$2.3\times10^{-13}$ between two different datasets	§6.6
$\mathrm{d}A/\mathrm{d}\theta$ vs sigmoid	$4\times10^{-11}$	§6.6
missing Jacobian	integral $1.647$; shape error $0.5275$	§6.7
mode of $\exp(X)$	$0.3679$, not $1$	§6.7
$P(\text{bag 2}\mid\text{mango})$	$\tfrac13$	Ex 6.4
Kalman trace reduction	$1.1278 \to 0.5334$	Ex 6.5

Where each section is used later

Section	Used in
§6.1 probability space	the vocabulary for everything after
§6.2 pmf/pdf/cdf	§6.7’s transforms; every likelihood in Chapters 9–12
§6.3 sum, product, Bayes	Chapter 8’s probabilistic modelling; all inference
§6.4 mean, covariance	Chapter 10’s PCA (the covariance matrix is PCA)
§6.5 Gaussian	Chapter 9’s regression, Chapter 11’s mixtures, Kalman filters
§6.6 conjugacy, exp family	Chapter 9’s Bayesian regression, Chapter 12’s losses
§6.7 change of variables	normalising flows, reparameterised gradients

The eight mistakes this chapter prevents

1. Reading a density as a probability. Definition 6.1 bounds the area, not the height; $39.89$ is a perfectly ordinary density value.
2. Forgetting the pre-image. Equation 6.8 sums over every outcome mapping to a state; skip it and Example 6.1’s $0.42$ becomes $0.21$.
3. Transposing a conditional. Equation 6.13 normalises columns, 6.14 rows. Both look right; only the axis sums distinguish them.
4. Confusing $p(y\mid x)$ with $p(x\mid y)$. Measured on a medical test: $0.99$ against $0.0902$.
5. Using Equation 6.44 on offset data. Returns $0$ for a variance of $4$ at an offset of $10^9$.
6. Concluding independence from zero covariance. Example 6.5 is a deterministic relationship with zero covariance.
7. Averaging component variances for a mixture. Equation 6.82’s second term was $90\%$ of the answer.
8. Dropping the Jacobian in a change of variables, or trusting a mode across a reparameterisation.

Every NumPy call in the chapter

Task	Call
joint from counts	n_ij / n_ij.sum()
marginal	joint.sum(axis=k) — sum the axis you remove
conditional on $X$	joint / joint.sum(axis=0)[None, :]
variance, safely	np.var(x) (two-pass internally), never mean(x**2) - mean(x)**2
unbiased variance	np.var(x, ddof=1)
covariance matrix	np.cov(X.T, bias=True) for the book’s $1/N$
correlation	np.corrcoef(x, y)
Gaussian conditioning	mu_x + Sxy @ np.linalg.solve(Syy, y - mu_y)
sample a Gaussian	z @ np.linalg.cholesky(S).T + mu
log-density, stably	np.linalg.slogdet plus np.linalg.solve, never inv
Beta functions	math.lgamma, and work in logs
inverse-transform sample	F_inverse(rng.random(n))

pch.quizTag Check your understanding

1. Which of the chapter's results is the computational bottleneck in probabilistic modelling?

   A Bayes' theorem, Equation 6.23 B The sum rule, Equation 6.20 — marginalising D binary variables costs 2^D terms, which is 1.27e30 at D = 100, with no known exact polynomial-time algorithm C The product rule, Equation 6.22 D The change of variables, Equation 6.143
   pch.quizShowAnswer

   B — The sum rule, Equation 6.20 — marginalising D binary variables costs 2^D terms, which is 1.27e30 at D = 100, with no known exact polynomial-time algorithm — The book flags it in a Remark. Bayes and the product rule are free; the evidence in Equation 6.27 is a high-dimensional integral, which is why variational inference and MCMC exist at all.

2. What single property makes the Gaussian the workhorse of the chapter?

   A It arises from the central limit theorem B Closure: marginalising, conditioning, multiplying, adding and affine mapping all return Gaussians, each by a closed-form formula in the mean and covariance — so inference is matrix algebra rather than integration C It has the maximum entropy for a given variance D Its mode equals its mean
   pch.quizShowAnswer

   B — Closure: marginalising, conditioning, multiplying, adding and affine mapping all return Gaussians, each by a closed-form formula in the mean and covariance — so inference is matrix algebra rather than integration — That is why Exercise 6.5 produces the Kalman filter with no new mathematics, and why Exercise 6.12 produces Bayesian linear regression the same way.

3. Why is conjugacy more than a convenience?

   A It gives a more accurate posterior B It satisfies the second desideratum: the parameter count does not grow with the data. Measured at two numbers after 100000 observations, against 200^D for a grid — 1e23 at D = 10 C It removes the need for a prior D It makes the likelihood normalisable
   pch.quizShowAnswer

   B — It satisfies the second desideratum: the parameter count does not grow with the data. Measured at two numbers after 100000 observations, against 200^D for a grid — 1e23 at D = 10 — And it works because multiplying exponential-family densities adds natural parameters. Pitman, Darmois and Koopman proved those are the only families with finite-dimensional sufficient statistics.

4. You reparameterise a model from sigma to log-sigma and your MAP estimate changes. Bug or not?

   A A bug in the optimiser B Not a bug. A density's mode is not reparameterisation-invariant, because the Jacobian reweights it — measured, the mode of exp(X) is 0.3679 rather than exp(0) = 1. Quantiles, including the median, ARE invariant C A bug in the prior D Only an issue for improper priors
   pch.quizShowAnswer

   B — Not a bug. A density's mode is not reparameterisation-invariant, because the Jacobian reweights it — measured, the mode of exp(X) is 0.3679 rather than exp(0) = 1. Quantiles, including the median, ARE invariant — Only affine reparameterisations leave the mode alone, since their Jacobian is constant. If a point estimate must survive a change of coordinates, report a posterior quantile rather than a MAP.

5. Two datasets give identical Gaussian log-likelihoods to 2.3e-13 but look completely different. What follows?

   A One of them must be corrupted B They share the sufficient statistics — n, the sum and the sum of squares — so by Theorem 6.14 no inference about mu or sigma can distinguish them. Everything downstream of the likelihood inherits that blindness C The likelihood was computed wrongly D The difference will show up at larger sample sizes
   pch.quizShowAnswer

   B — They share the sufficient statistics — n, the sum and the sum of squares — so by Theorem 6.14 no inference about mu or sigma can distinguish them. Everything downstream of the likelihood inherits that blindness — This is the flip side of conjugacy's convenience: a fixed parameter count can absorb unbounded data precisely because the data compresses to a fixed-size summary first. Skew and multimodality are simply not in that summary.

Recall card

• Two rules generate the chapter. The sum rule (Eq 6.20) collapses an axis; the product rule (Eq 6.22) factorises a cell. Bayes (Eq 6.23) is those two rearranged.
• The sum rule is the expensive one: 2^D terms for D binary variables, 1.27e30 at D = 100, with no known exact polynomial-time algorithm.
• A random variable is a FUNCTION, and Eq 6.8 defines its law through the pre-image because that function can collapse outcomes.
• A density is not a probability. Def 6.1 bounds the integral, not the height — measured 1.4286, 39.89 and 1e8 with the integral exactly 1 each time. And P(X=x) = 0 exactly for continuous X.
• The cdf is the only one of the three that is always a probability, and some cdfs have no pdf.
• Eq 6.13 normalises columns, Eq 6.14 rows. Assert the axis sums; a transposed conditional is invisible otherwise.
• The likelihood is not a distribution in x. It needs the prior and the evidence to become one.
• The evidence (Eq 6.27) is the likelihood averaged under the PRIOR, and it is the hard integral.
• A zero prior is permanent — exactly 0.0000000000 after 400 confirming observations, silently.
• The base-rate fallacy, measured: P(disease | +) = 0.0902 at prevalence 0.001 with a 99% test, an overstatement of 10.98x.
• Expectation is linear (Eq 6.34) with no independence needed; covariance (Eq 6.36) measures only LINEAR dependence, so zero covariance does not imply independence.
• Eq 6.44 is numerically unsafe: returns 0 for a variance of 4 once the data is offset by 1e9. Use Eq 6.43.
• The book’s empirical covariance is BIASED (1/N) and it says so; at N = 2 that is half the true variance.
• Eq 6.51’s V[Ax+b] = A Sigma A-transpose takes the OUTPUT’s shape and can be singular, hence have no density.
• The Gaussian is closed under everything — marginalise (6.68), condition (6.66, 6.67), multiply (6.74–6.77), add (6.78), map affinely (6.88) — each in closed form.
• Conditioning can only shrink the covariance, and not at all if the cross-covariance is zero.
• Precisions ADD when Gaussians multiply, so a Gaussian posterior is always sharper than its prior.
• A mixture of Gaussians is NOT Gaussian (Thm 6.12), and its variance has a between-component term that was 90% of the answer in the worked case.
• Sample a Gaussian with the Cholesky factor: triangular, and guaranteed to exist for a covariance matrix.
• Eq 6.107’s exponential family has sufficiency built in — the data reaches the parameters only through phi(x) — and dA/dtheta = E[phi(x)].
• The Bernoulli’s natural parameter is the log-odds, and its inverse is the SIGMOID (Eq 6.118).
• Conjugacy is structural: multiplying exponential-family densities adds natural parameters, so Eq 6.120 generates a conjugate prior for every member.
• Transforming a continuous variable needs a Jacobian (Eq 6.143, Thm 6.16); dropping it is a shape error no renormalisation repairs.
• Theorem 6.15: F_X(X) is uniform for any continuous X with a strictly monotonic cdf — moments are irrelevant. Backwards, it is inverse-transform sampling.
• A mode is not reparameterisation-invariant; a quantile is.

Next: Probability and Distributions Overview for the chapter map, or on to Chapter 7, Continuous Optimization.