Chapter 5 Exercises and Solutions

Nine exercises, worked in full. Every derivative below is checked against a central difference at several points, and every gap is reported — they run from 0.0e+00 to 1.1e-09, which is the differencing error rather than an error in the answer.

Six of the nine are really one question asked six ways: what are the dimensions, and in what order do the factors multiply? If you can state the shape of every partial derivative before computing any of them, the rest is bookkeeping. That is why Exercises 5.7 and 5.8 explicitly ask for the dimensions.

How to use this page

Do 5.1 to 5.4 with a pen; they are one-line answers and the only trap is simplifying too late. 5.5 to 5.8 are the chapter’s real content — for each one, write down the shape of every factor first, then multiply. 5.9 is the one worth sitting with: it is short, it is the gradient behind variational autoencoders, and the obvious answer is wrong by 102%.

A convention that matters throughout

Following §5.2’s Equation 5.40, the derivative of a scalar function of $\mathbf{x} \in \mathbb{R}^D$ is a row vector, $1 \times D$. So $\mathrm{d}f/\mathrm{d}\mathbf{x}$ for $f(\mathbf{x}) = \mathbf{x}^\top\mathbf{x}$ is $2\mathbf{x}^\top$, not $2\mathbf{x}$. Keeping to that makes every chain rule below a plain matrix product with the shapes lining up left to right, and no transposes to remember.

5.1 — A product of two composed functions

Compute $f'(x)$ for

$$f(x) = \log(x^4)\sin(x^3)$$

Simplify before differentiating. $\log(x^4) = 4\log|x|$. That single step turns a chain rule on a fourth power into a constant times a logarithm, and it is where both numbers in the answer come from.

Now the product rule, with $u = 4\log|x|$ and $v = \sin(x^3)$:

$$u' = \frac{4}{x}, \qquad v' = 3x^2\cos(x^3)$$

so

$$f'(x) = \frac{4}{x}\sin(x^3) + 4\log|x| \cdot 3x^2\cos(x^3) = \boxed{\frac{4\sin(x^3)}{x} + 12x^2\log|x|\cos(x^3)}$$

Checked at five points against a central difference; worst gap $1.1 \times 10^{-9}$.

Note what happens at x = 1

$\log(1) = 0$, so the second term vanishes and $f'(1) = 4\sin(1) = 3.365884$. A point where one term of a two-term derivative is exactly zero is a good sanity check and a bad test case — it would not catch an error in the term that disappeared.

5.2 — The logistic sigmoid

Compute $f'(x)$ for

$$f(x) = \frac{1}{1 + \exp(-x)}$$

Write it as $(1 + e^{-x})^{-1}$ and apply the chain rule:

$$f'(x) = -(1 + e^{-x})^{-2}\cdot\frac{\mathrm{d}}{\mathrm{d}x}\left(1 + e^{-x}\right) = -(1 + e^{-x})^{-2}\cdot(-e^{-x}) = \frac{e^{-x}}{(1 + e^{-x})^2}$$

Now the form worth memorising. Split the fraction:

$$\frac{e^{-x}}{(1+e^{-x})^2} = \frac{1}{1+e^{-x}}\cdot\frac{e^{-x}}{1+e^{-x}} = \frac{1}{1+e^{-x}}\left(1 - \frac{1}{1+e^{-x}}\right)$$

using $\dfrac{e^{-x}}{1+e^{-x}} = \dfrac{(1 + e^{-x}) - 1}{1+e^{-x}} = 1 - f(x)$. So

$$\boxed{f'(x) = f(x)\bigl(1 - f(x)\bigr)}$$

The derivative is a function of the output, which is why a sigmoid layer’s backward pass needs no extra evaluation of the exponential — it reuses the activation it already stored.

$x$	$f(1-f)$	$e^{-x}/(1+e^{-x})^2$	central difference
$-3$	$0.0451766597$	$0.0451766597$	$0.0451766597$
$-1$	$0.1966119332$	$0.1966119332$	$0.1966119332$
$0$	$0.2500000000$	$0.2500000000$	$0.2500000000$
$1$	$0.1966119332$	$0.1966119332$	$0.1966119332$
$3$	$0.0451766597$	$0.0451766597$	$0.0451766597$

The number to remember is $\tfrac{1}{4}$. $f(1-f)$ is maximised where $f = \tfrac{1}{2}$, at $x = 0$, and its value there is exactly $0.25$. Every sigmoid layer multiplies the backward signal by at most $\tfrac{1}{4}$, so ten stacked sigmoids attenuate the gradient by at least $4^{-10} \approx 10^{-6}$ — which is the vanishing gradient problem as a one-line calculation, and the reason ReLU (derivative exactly $1$ where active) replaced it.

5.3 — An unnormalised Gaussian

Compute $f'(x)$ for

$$f(x) = \exp\!\left(-\frac{1}{2\sigma^2}(x-\mu)^2\right), \qquad \mu, \sigma \in \mathbb{R} \text{ constant}$$

Chain rule with the exponent as the inner function. Its derivative is $-\frac{1}{2\sigma^2}\cdot 2(x - \mu) = -\frac{x-\mu}{\sigma^2}$, so

$$\boxed{f'(x) = -\frac{x - \mu}{\sigma^2}\,f(x)}$$

The derivative is again a multiple of the function itself — the defining property of the exponential, and the reason this one is easy while $\exp(-x^2)$‘s antiderivative has no elementary form at all.

$x$	$f'(x)$ analytic	central difference
$-0.50$	$0.1346621949$	$0.1346621949$
$0.60$	$0.8664723710$	$0.8664723710$
$1.30 = \mu$	$-0.0000000000$	$0.0000000000$
$2.00$	$-0.8664723710$	$-0.8664723710$
$3.10$	$-0.1346621949$	$-0.1346621950$

(with $\mu = 1.3$, $\sigma = 0.7$). At $x = \mu$ the derivative is exactly zero — $f$ has one stationary point and it is the peak, since $f > 0$ everywhere. Note also the antisymmetry about $\mu$: $f'(\mu - t) = -f'(\mu + t)$, visible in rows 2 and 4 and rows 1 and 5.

5.4 — Taylor polynomials of sin plus cos

Compute $T_n$ for $n = 0, \ldots, 5$ of $f(x) = \sin(x) + \cos(x)$ at $x_0 = 0$.

Do not differentiate six times. Note that

$$\sin x + \cos x = \sqrt{2}\,\sin\!\left(x + \tfrac{\pi}{4}\right)$$

so the derivatives must cycle with period 4. Confirm on the first few:

$k$	$f^{(k)}(x)$	$f^{(k)}(0)$
$0$	$\sin x + \cos x$	$1$
$1$	$\cos x - \sin x$	$1$
$2$	$-\sin x - \cos x$	$-1$
$3$	$-\cos x + \sin x$	$-1$
$4$	$\sin x + \cos x$	$1$
$5$	$\cos x - \sin x$	$1$

The pattern $+1, +1, -1, -1$ repeats forever. Divide by $k!$ (Definition 5.8) and accumulate:

$$T_0(x) = 1$$ $$T_1(x) = 1 + x$$ $$T_2(x) = 1 + x - \frac{x^2}{2}$$ $$T_3(x) = 1 + x - \frac{x^2}{2} - \frac{x^3}{6}$$ $$T_4(x) = 1 + x - \frac{x^2}{2} - \frac{x^3}{6} + \frac{x^4}{24}$$ $$T_5(x) = 1 + x - \frac{x^2}{2} - \frac{x^3}{6} + \frac{x^4}{24} + \frac{x^5}{120}$$

which is exactly the sum of $\sin$‘s series $x - \frac{x^3}{6} + \frac{x^5}{120}$ and $\cos$‘s $1 - \frac{x^2}{2} + \frac{x^4}{24}$ — as it must be, since Taylor expansion is linear.

The error table is the interesting part:

$x$	$T_0$	$T_1$	$T_2$	$T_3$	$T_4$	$T_5$
$0.1$	$9.48\text{e-}02$	$5.16\text{e-}03$	$1.62\text{e-}04$	$4.25\text{e-}06$	$8.19\text{e-}08$	$1.41\text{e-}09$
$0.5$	$3.57\text{e-}01$	$1.43\text{e-}01$	$1.80\text{e-}02$	$2.84\text{e-}03$	$2.37\text{e-}04$	$2.31\text{e-}05$
$1.0$	$3.82\text{e-}01$	$6.18\text{e-}01$	$1.18\text{e-}01$	$4.84\text{e-}02$	$6.77\text{e-}03$	$1.56\text{e-}03$
$2.0$	$5.07\text{e-}01$	$2.51\text{e+}00$	$5.07\text{e-}01$	$8.26\text{e-}01$	$1.60\text{e-}01$	$1.07\text{e-}01$
$4.0$	$2.41\text{e+}00$	$6.41\text{e+}00$	$1.59\text{e+}00$	$1.23\text{e+}01$	$1.59\text{e+}00$	$6.94\text{e+}00$

Read it two ways. Down a column the error collapses — at $x = 0.1$, from $9.5\text{e-}02$ to $1.4\text{e-}09$. Across a row it is not even monotone: at $x = 2$, $T_3$‘s error of $0.826$ is worse than $T_2$‘s $0.507$; at $x = 4$ the ranking is scrambled entirely ($T_2$ at $1.59$, $T_3$ at $12.3$, $T_4$ at $1.59$, $T_5$ at $6.94$). More degrees is a promise about the limit $x \to x_0$, not about any particular $x$.

5.5 — Dimensions and Jacobians of three functions

$$f_1(\mathbf{x}) = \sin(x_1)\cos(x_2), \quad \mathbf{x} \in \mathbb{R}^2$$ $$f_2(\mathbf{x}, \mathbf{y}) = \mathbf{x}^\top\mathbf{y}, \quad \mathbf{x}, \mathbf{y} \in \mathbb{R}^n$$ $$f_3(\mathbf{x}) = \mathbf{x}\mathbf{x}^\top, \quad \mathbf{x} \in \mathbb{R}^n$$

a. The dimensions

The rule is mechanical: the derivative’s shape is the output’s shape followed by the input’s shape.

	output	input	$\partial f_i / \partial \mathbf{x}$
$f_1$	scalar	$\mathbb{R}^2$	$1 \times 2$
$f_2$	scalar	$\mathbb{R}^n$	$1 \times n$ (and $1 \times n$ for $\mathbf{y}$)
$f_3$	$\mathbb{R}^{n\times n}$	$\mathbb{R}^n$	$n \times n \times n$ — a third-order tensor

$f_3$ is the one that catches people. Its output is a matrix, so its derivative gains an index and stops being a matrix.

b. The Jacobians

$f_1$. Two partials, each treating the other variable as constant:

$$\frac{\partial f_1}{\partial \mathbf{x}} = \begin{bmatrix} \cos(x_1)\cos(x_2) & -\sin(x_1)\sin(x_2)\end{bmatrix} \in \mathbb{R}^{1\times 2}$$

$f_2$. $\mathbf{x}^\top\mathbf{y} = \sum_i x_i y_i$, so $\partial/\partial x_i = y_i$:

$$\frac{\partial f_2}{\partial \mathbf{x}} = \mathbf{y}^\top \in \mathbb{R}^{1\times n}, \qquad \frac{\partial f_2}{\partial \mathbf{y}} = \mathbf{x}^\top \in \mathbb{R}^{1\times n}$$

$f_3$. Write the entry out: $(f_3)_{ij} = x_i x_j$. Differentiate with respect to $x_k$ by the product rule, and each factor contributes only when its index matches:

$$\frac{\partial (f_3)_{ij}}{\partial x_k} = \delta_{ik}x_j + x_i\delta_{jk}$$

where $\delta$ is the Kronecker delta. Checked against differencing the matrix-valued function: gap $1.8 \times 10^{-10}$, and of the $64$ entries at $n = 4$, $28$ are nonzero — the tensor is sparse, but not sparse enough to be worth storing for large $n$.

Why writing out the entry beats matrix rules

There is no clean “matrix calculus identity” for $\partial(\mathbf{x}\mathbf{x}^\top)/\partial\mathbf{x}$, because the result is not a matrix. Dropping to indices — $(f_3)_{ij} = x_ix_j$, differentiate, collect Kroneckers — always works and never needs a lookup table. §5.4’s advice to “differentiate entry by entry” is not a beginner’s crutch; for tensor-valued derivatives it is the only method.

5.6 — A composed scalar, and a trace

Differentiate $f$ with respect to $\mathbf{t}$ and $g$ with respect to $\mathbf{X}$:

$$f(\mathbf{t}) = \sin\!\left(\log(\mathbf{t}^\top\mathbf{t})\right), \quad \mathbf{t} \in \mathbb{R}^D$$ $$g(\mathbf{X}) = \operatorname{tr}(\mathbf{A}\mathbf{X}\mathbf{B}), \quad \mathbf{A} \in \mathbb{R}^{D\times E},\ \mathbf{X} \in \mathbb{R}^{E\times F},\ \mathbf{B}\in\mathbb{R}^{F\times D}$$

$f$: three links, shapes stated first. Let $u = \mathbf{t}^\top\mathbf{t}$ and $v = \log u$.

link	derivative	shape
$\sin$	$\cos(v)$	$1 \times 1$
$\log$	$1/u$	$1 \times 1$
$\mathbf{t}^\top\mathbf{t}$	$2\mathbf{t}^\top$	$1 \times D$

Multiply left to right:

$$\boxed{\frac{\mathrm{d}f}{\mathrm{d}\mathbf{t}} = \cos\!\left(\log(\mathbf{t}^\top\mathbf{t})\right)\cdot\frac{2\mathbf{t}^\top}{\mathbf{t}^\top\mathbf{t}} \in \mathbb{R}^{1\times D}}$$

Gap against differencing: $2.6 \times 10^{-11}$.

$g$: rearrange the trace first. The trace is invariant under cyclic permutation, so

$$\operatorname{tr}(\mathbf{A}\mathbf{X}\mathbf{B}) = \operatorname{tr}(\mathbf{B}\mathbf{A}\mathbf{X})$$

Verified numerically: both equal $-6.1094555522$. Now $\operatorname{tr}(\mathbf{M}\mathbf{X}) = \sum_{i,j} M_{ij}X_{ji}$, so $\partial/\partial X_{ji} = M_{ij}$, which is

$$\boxed{\frac{\mathrm{d}g}{\mathrm{d}\mathbf{X}} = (\mathbf{B}\mathbf{A})^\top \in \mathbb{R}^{E\times F}}$$

Check the shape. $\mathbf{B}\mathbf{A}$ is $(F\times D)(D\times E) = F \times E$, so its transpose is $E \times F$ — the shape of $\mathbf{X}$, which it must be for a scalar function of a matrix. Gap against differencing every entry: $8.9\times10^{-10}$.

That cyclic rearrangement is the whole exercise. Attacked directly, $\operatorname{tr}(\mathbf{A}\mathbf{X}\mathbf{B})$ is a triple sum; rearranged, it is one line.

5.7 — Two chain rules, with every dimension

a.

$$f(z) = \log(1 + z), \qquad z = \mathbf{x}^\top\mathbf{x}, \qquad \mathbf{x} \in \mathbb{R}^D$$

step	value	shape
$\partial f/\partial z$	$\dfrac{1}{1+z}$	$1 \times 1$
$\partial z/\partial \mathbf{x}$	$2\mathbf{x}^\top$	$1 \times D$
product	$\dfrac{2\mathbf{x}^\top}{1 + \mathbf{x}^\top\mathbf{x}}$	$1 \times D$

$$\boxed{\frac{\mathrm{d}f}{\mathrm{d}\mathbf{x}} = \frac{2\mathbf{x}^\top}{1 + \mathbf{x}^\top\mathbf{x}}}$$

Gap: $5.5\times10^{-11}$. Note $1 + z > 0$ always, since $z$ is a squared norm — so this derivative never blows up, which is one reason $\log(1 + \mathbf{x}^\top\mathbf{x})$ shows up as a well-behaved penalty.

b.

$$f(\mathbf{z}) = \sin(\mathbf{z}) \text{ elementwise}, \qquad \mathbf{z} = \mathbf{A}\mathbf{x} + \mathbf{b}, \qquad \mathbf{A} \in \mathbb{R}^{E\times D},\ \mathbf{x}\in\mathbb{R}^D,\ \mathbf{b}\in\mathbb{R}^E$$

step	value	shape
$\partial f/\partial \mathbf{z}$	$\operatorname{diag}(\cos\mathbf{z})$	$E \times E$
$\partial \mathbf{z}/\partial \mathbf{x}$	$\mathbf{A}$	$E \times D$
product	$\operatorname{diag}(\cos(\mathbf{A}\mathbf{x}+\mathbf{b}))\,\mathbf{A}$	$E \times D$

$$\boxed{\frac{\mathrm{d}f}{\mathrm{d}\mathbf{x}} = \operatorname{diag}\bigl(\cos(\mathbf{A}\mathbf{x}+\mathbf{b})\bigr)\mathbf{A} \in \mathbb{R}^{E\times D}}$$

Gap: $1.1\times10^{-10}$. The Jacobian of an elementwise function is diagonal — output $i$ depends only on input $i$ — which is why every activation function in a network contributes a diagonal factor and never a dense one.

Never build the diagonal matrix

$\operatorname{diag}(\cos\mathbf{z})\mathbf{A}$ scales row $i$ of $\mathbf{A}$ by $\cos z_i$. Writing it as a matrix product costs $E^2$ storage and $E^2D$ multiplications to achieve $ED$ multiplications’ worth of work. In code it is np.cos(z)[:, None] * A. At $E = 4096$ that is the difference between a broadcast and a 16-million-entry allocation, and it is why frameworks store activation derivatives as vectors.

5.8 — Three more, including one with a trap

a.

$$f(z) = \exp\!\left(-\tfrac{1}{2}z\right), \qquad z = g(\mathbf{y}) = \mathbf{y}^\top\mathbf{S}^{-1}\mathbf{y}, \qquad \mathbf{y} = h(\mathbf{x}) = \mathbf{x} - \boldsymbol{\mu}$$

with $\mathbf{x}, \boldsymbol{\mu} \in \mathbb{R}^D$ and $\mathbf{S} \in \mathbb{R}^{D\times D}$.

step	value	shape
$\partial f/\partial z$	$-\tfrac{1}{2}\exp(-\tfrac{1}{2}z)$	$1 \times 1$
$\partial z/\partial \mathbf{y}$	$\mathbf{y}^\top(\mathbf{S}^{-1} + \mathbf{S}^{-\top})$	$1 \times D$
$\partial \mathbf{y}/\partial \mathbf{x}$	$\mathbf{I}$	$D \times D$

The middle row is the trap. For a general $\mathbf{S}$ the derivative of a quadratic form $\mathbf{y}^\top\mathbf{M}\mathbf{y}$ is $\mathbf{y}^\top(\mathbf{M} + \mathbf{M}^\top)$, not $2\mathbf{y}^\top\mathbf{M}$. The two agree exactly when $\mathbf{M}$ is symmetric — which it is here whenever $\mathbf{S}$ is a covariance matrix, and that is the only case the book has in mind. Under that assumption:

$$\boxed{\frac{\mathrm{d}f}{\mathrm{d}\mathbf{x}} = -\exp\!\left(-\tfrac{1}{2}\mathbf{y}^\top\mathbf{S}^{-1}\mathbf{y}\right)\mathbf{y}^\top\mathbf{S}^{-1} \in \mathbb{R}^{1\times D}}$$

(the $\tfrac{1}{2}$ and the $2$ cancel). Both the general and the symmetric form check out to $4.9\times10^{-11}$ on a symmetric $\mathbf{S}$.

What the shortcut costs when $\mathbf{S}$ is not symmetric. Measured on a deliberately non-symmetric $\mathbf{S}$: the general form is still right to $2.7\times10^{-11}$, and the shortcut $-\exp(-z/2)\mathbf{y}^\top\mathbf{S}^{-1}$ is off by $1.4\times10^{-1}$ — 62.5% relative. Not a rounding difference: a wrong answer with a plausible shape.

This is $\log$ of a Gaussian density differentiated, so the boxed formula is the score function you use in maximum likelihood, in Mahalanobis distances, and in Chapter 6.

b.

$$f(\mathbf{x}) = \operatorname{tr}(\mathbf{x}\mathbf{x}^\top + \sigma^2\mathbf{I}), \qquad \mathbf{x} \in \mathbb{R}^D$$

The book’s hint is explicitly write out the outer product — take it. The $(i,i)$ entry of $\mathbf{x}\mathbf{x}^\top$ is $x_i^2$, and the $(i,i)$ entry of $\sigma^2\mathbf{I}$ is $\sigma^2$, so

$$f(\mathbf{x}) = \sum_{i=1}^{D}\left(x_i^2 + \sigma^2\right) = \mathbf{x}^\top\mathbf{x} + D\sigma^2$$

There was never a matrix here. Differentiate the scalar:

$$\boxed{\frac{\mathrm{d}f}{\mathrm{d}\mathbf{x}} = 2\mathbf{x}^\top \in \mathbb{R}^{1\times D}}$$

Verified: at $D = 4$, $\sigma^2 = 0.75$, the trace evaluates to $5.90317933$ and so does $\mathbf{x}^\top\mathbf{x} + D\sigma^2$; the derivative matches differencing to $4.8\times10^{-10}$. Note $\sigma^2$ contributes $3.00$ to the value and exactly nothing to the derivative — a constant offset that many people carry through three lines of matrix algebra before it cancels.

c.

$$\mathbf{f} = \tanh(\mathbf{z}) \in \mathbb{R}^M, \qquad \mathbf{z} = \mathbf{A}\mathbf{x} + \mathbf{b}, \qquad \mathbf{x}\in\mathbb{R}^N,\ \mathbf{A}\in\mathbb{R}^{M\times N},\ \mathbf{b}\in\mathbb{R}^M$$

The exercise says the product need not be computed explicitly — only the dimensions. Using $\tanh' = 1 - \tanh^2$:

step	value	shape
$\partial \mathbf{f}/\partial \mathbf{z}$	$\operatorname{diag}(1 - \tanh^2\mathbf{z})$	$M \times M$
$\partial \mathbf{z}/\partial \mathbf{x}$	$\mathbf{A}$	$M \times N$
product	$\operatorname{diag}(1-\tanh^2\mathbf{z})\,\mathbf{A}$	$M \times N$

$$\boxed{\frac{\mathrm{d}\mathbf{f}}{\mathrm{d}\mathbf{x}} = \operatorname{diag}\bigl(1 - \tanh^2(\mathbf{A}\mathbf{x}+\mathbf{b})\bigr)\mathbf{A} \in \mathbb{R}^{M\times N}}$$

Gap: $7.8\times10^{-11}$. This is one layer of a neural network, and its Jacobian is the object §5.6’s Equation 5.118 multiplies together $K$ times. Every diagonal entry lies in $(0, 1]$ — measured maximum here $0.692457$ — and that is where vanishing gradients come from: a product of factors each no larger than $1$.

5.9 — The reparameterisation gradient

Define

$$g(\mathbf{z}, \boldsymbol{\nu}) := \log p(\mathbf{x}, \mathbf{z}) - \log q(\mathbf{z}, \boldsymbol{\nu}), \qquad \mathbf{z} := t(\boldsymbol{\epsilon}, \boldsymbol{\nu})$$

for differentiable $p$, $q$, $t$. Compute $\dfrac{\mathrm{d}}{\mathrm{d}\boldsymbol{\nu}}g(\mathbf{z}, \boldsymbol{\nu})$.

The whole exercise is noticing that $\boldsymbol{\nu}$ appears twice. It is inside $\mathbf{z} = t(\boldsymbol{\epsilon}, \boldsymbol{\nu})$, and it is again a direct argument of $\log q$. A total derivative must follow both paths:

$$\frac{\mathrm{d}g}{\mathrm{d}\boldsymbol{\nu}} = \underbrace{\frac{\partial g}{\partial \mathbf{z}}\frac{\partial \mathbf{z}}{\partial \boldsymbol{\nu}}}_{\text{through } \mathbf{z}} + \underbrace{\frac{\partial g}{\partial \boldsymbol{\nu}}}_{\text{directly}}$$

Expanding each piece:

$$\frac{\partial g}{\partial \mathbf{z}} = \frac{\partial \log p(\mathbf{x},\mathbf{z})}{\partial \mathbf{z}} - \frac{\partial \log q(\mathbf{z},\boldsymbol{\nu})}{\partial \mathbf{z}}, \qquad \frac{\partial \mathbf{z}}{\partial \boldsymbol{\nu}} = \frac{\partial t(\boldsymbol{\epsilon},\boldsymbol{\nu})}{\partial \boldsymbol{\nu}}, \qquad \frac{\partial g}{\partial \boldsymbol{\nu}} = -\frac{\partial \log q(\mathbf{z},\boldsymbol{\nu})}{\partial \boldsymbol{\nu}}$$

so

$$\boxed{\frac{\mathrm{d}g}{\mathrm{d}\boldsymbol{\nu}} = \left(\frac{\partial \log p(\mathbf{x},\mathbf{z})}{\partial \mathbf{z}} - \frac{\partial \log q(\mathbf{z},\boldsymbol{\nu})}{\partial \mathbf{z}}\right)\frac{\partial t(\boldsymbol{\epsilon},\boldsymbol{\nu})}{\partial \boldsymbol{\nu}} - \frac{\partial \log q(\mathbf{z},\boldsymbol{\nu})}{\partial \boldsymbol{\nu}}}$$

Where the $\partial \log q/\partial\boldsymbol{\nu}$ in the second term is evaluated at fixed $\mathbf{z}$, while the one in the bracket is $\partial/\partial\mathbf{z}$ at fixed $\boldsymbol{\nu}$. Same function, two different partials, and they do not cancel.

What this is, and what dropping a term costs

$g$ is the integrand of the evidence lower bound, $\boldsymbol{\nu}$ are the parameters of a variational distribution $q$, and $\mathbf{z} = t(\boldsymbol{\epsilon}, \boldsymbol{\nu})$ is the reparameterisation trick: sample a fixed noise $\boldsymbol{\epsilon}$, then push it through a $\boldsymbol{\nu}$-dependent transform so that the sampling step has no gradient of its own. This exercise is the gradient that trains a variational autoencoder.

Instantiated concretely — $q$ Gaussian with $t(\epsilon, \boldsymbol{\nu}) = \nu_1 + e^{\nu_2}\epsilon$, and $p$ a product of two Gaussians:

	$\partial/\partial\nu_1$	$\partial/\partial\nu_2$
finite differences (the truth)	$-0.304384$	$0.877579$
chain rule, both paths	$-0.304384$	$0.877579$
only the path through $\mathbf{z}$	$+0.590711$	$0.237579$

Gap for the correct form: $1.4\times10^{-10}$. Gap for the truncated one: $0.895095$, which is 102% relative — and note the sign flip on the first component. A gradient that points the wrong way does not converge slowly, it diverges.

The direct path is not optional

It is tempting to reason: ”$\log q$ is the entropy term, its expectation over $q$ is a constant, so its gradient vanishes.” The expectation argument is about $\mathbb{E}_q[\partial \log q/\partial\boldsymbol{\nu}]$, which is indeed zero — that is the score-function identity. But you are differentiating a single sample, not an expectation, and pointwise the term is emphatically nonzero. Here it is the difference between $-0.304$ and $+0.591$.

All nine, in one table

	Function	Answer	Shape	Gap vs differences
5.1	$\log(x^4)\sin(x^3)$	$\dfrac{4\sin(x^3)}{x} + 12x^2\log\lvert x\rvert\cos(x^3)$	scalar	$1.1\text{e-}09$
5.2	$\bigl(1+e^{-x}\bigr)^{-1}$	$f(1-f)$, max $\tfrac{1}{4}$ at $0$	scalar	$6.3\text{e-}11$
5.3	$\exp\!\left(-\tfrac{(x-\mu)^2}{2\sigma^2}\right)$	$-\dfrac{x-\mu}{\sigma^2}f(x)$	scalar	$2.7\text{e-}11$
5.4	$\sin x + \cos x$ at $0$	coefficients cycle $+1,+1,-1,-1$	scalar	—
5.5a	—	$1\times2$, $1\times n$, $n\times n\times n$	—	—
5.5b	$\mathbf{x}\mathbf{x}^\top$	$\delta_{ik}x_j + x_i\delta_{jk}$	$n\times n\times n$	$1.8\text{e-}10$
5.6	$\sin(\log \mathbf{t}^\top\mathbf{t})$	$\cos(\log\mathbf{t}^\top\mathbf{t})\dfrac{2\mathbf{t}^\top}{\mathbf{t}^\top\mathbf{t}}$	$1\times D$	$2.6\text{e-}11$
5.6	$\operatorname{tr}(\mathbf{A}\mathbf{X}\mathbf{B})$	$(\mathbf{B}\mathbf{A})^\top$	$E\times F$	$8.9\text{e-}10$
5.7a	$\log(1+\mathbf{x}^\top\mathbf{x})$	$\dfrac{2\mathbf{x}^\top}{1+\mathbf{x}^\top\mathbf{x}}$	$1\times D$	$5.5\text{e-}11$
5.7b	$\sin(\mathbf{A}\mathbf{x}+\mathbf{b})$	$\operatorname{diag}(\cos\mathbf{z})\mathbf{A}$	$E\times D$	$1.1\text{e-}10$
5.8a	$\exp(-\tfrac{1}{2}\mathbf{y}^\top\mathbf{S}^{-1}\mathbf{y})$	$-f\cdot\mathbf{y}^\top\mathbf{S}^{-1}$ (symmetric $\mathbf{S}$)	$1\times D$	$4.9\text{e-}11$
5.8b	$\operatorname{tr}(\mathbf{x}\mathbf{x}^\top + \sigma^2\mathbf{I})$	$2\mathbf{x}^\top$	$1\times D$	$4.8\text{e-}10$
5.8c	$\tanh(\mathbf{A}\mathbf{x}+\mathbf{b})$	$\operatorname{diag}(1-\tanh^2\mathbf{z})\mathbf{A}$	$M\times N$	$7.8\text{e-}11$
5.9	the ELBO integrand	two paths to $\boldsymbol{\nu}$, both needed	$1\times\lvert\boldsymbol{\nu}\rvert$	$1.4\text{e-}10$

The complete check

chapter_5_exercises.py

"""Every exercise at the end of Chapter 5, checked against finite differences."""
 
import numpy as np
 
np.set_printoptions(precision=6, suppress=True, linewidth=150)
rng = np.random.default_rng(5)
 
 
def fd(f, x, h=1e-6):
    """Central difference of a scalar function of a scalar."""
    return (f(x + h) - f(x - h)) / (2 * h)
 
 
def jac(f, x, h=1e-6):
    """Numerical Jacobian: rows = outputs, cols = inputs."""
    x = np.atleast_1d(np.asarray(x, dtype=float))
    f0 = np.atleast_1d(f(x))
    J = np.zeros((f0.size, x.size))
    for j in range(x.size):
        e = np.zeros(x.size)
        e[j] = h
        J[:, j] = (np.atleast_1d(f(x + e)) - np.atleast_1d(f(x - e))).ravel() / (2 * h)
    return J
 
 
print("########## ex_5_1")
f = lambda x: np.log(x ** 4) * np.sin(x ** 3)
# f' = 4 sin(x^3)/x + 12 x^2 log(x) cos(x^3)
df = lambda x: 4 * np.sin(x ** 3) / x + 12 * x ** 2 * np.log(x) * np.cos(x ** 3)
print("f(x) = log(x^4) sin(x^3)")
print("f'(x) = 4 sin(x^3)/x + 12 x^2 log(x) cos(x^3)")
print(f"  {'x':>6}  {'analytic':>14}  {'central diff':>14}  {'gap':>9}")
for x in (0.4, 0.8, 1.0, 1.5, 2.0):
    print(f"  {x:>6.2f}  {df(x):>14.8f}  {fd(f, x):>14.8f}  {abs(df(x) - fd(f, x)):>9.1e}")
print("  note log(x^4) = 4 log|x| BEFORE differentiating: that is where the 4 and")
print("  the 12 = 4*3 come from, and it is much less error-prone than the chain rule")
print("  on the fourth power.")
 
print()
print("########## ex_5_2")
sig = lambda x: 1 / (1 + np.exp(-x))
print("f(x) = 1/(1 + exp(-x))")
print("f'(x) = exp(-x)/(1+exp(-x))^2 = f(x)(1 - f(x))")
print(f"  {'x':>6}  {'f(1-f)':>14}  {'exp form':>14}  {'central diff':>14}  {'gap':>9}")
for x in (-3.0, -1.0, 0.0, 1.0, 3.0):
    a = sig(x) * (1 - sig(x))
    b = np.exp(-x) / (1 + np.exp(-x)) ** 2
    print(f"  {x:>6.1f}  {a:>14.10f}  {b:>14.10f}  {fd(sig, x):>14.10f}"
          f"  {abs(a - fd(sig, x)):>9.1e}")
print(f"  the maximum of f' is at x = 0, where it equals {sig(0.0) * (1 - sig(0.0)):.4f} = 1/4 exactly.")
print("  That 1/4 is the per-layer ceiling behind vanishing gradients in sigmoid nets.")
 
print()
print("########## ex_5_3")
mu, sd = 1.3, 0.7
g = lambda x: np.exp(-((x - mu) ** 2) / (2 * sd ** 2))
dg = lambda x: -(x - mu) / sd ** 2 * g(x)
print(f"f(x) = exp(-(x-mu)^2 / (2 sigma^2))   with mu = {mu}, sigma = {sd}")
print("f'(x) = -(x - mu)/sigma^2 * f(x)")
print(f"  {'x':>6}  {'analytic':>14}  {'central diff':>14}  {'gap':>9}")
for x in (-0.5, 0.6, 1.3, 2.0, 3.1):
    print(f"  {x:>6.2f}  {dg(x):>14.10f}  {fd(g, x):>14.10f}  {abs(dg(x) - fd(g, x)):>9.1e}")
print(f"  f'(mu) = {dg(mu):.1e} exactly: the only stationary point is the peak.")
 
print()
print("########## ex_5_4")
h = lambda x: np.sin(x) + np.cos(x)
# Derivatives at 0 cycle with period 4: 1, 1, -1, -1, 1, 1
coeffs = [1.0, 1.0, -1.0, -1.0, 1.0, 1.0]
print("f(x) = sin(x) + cos(x) at x0 = 0")
print("  derivatives at 0 cycle with period 4:  1, 1, -1, -1, 1, 1, ...")
import math
for n in range(6):
    terms = " + ".join(
        f"{coeffs[k] / math.factorial(k):+.6g} x^{k}" for k in range(n + 1))
    print(f"  T{n}(x) = {terms}")
print()
print(f"  {'x':>6}  {'f(x)':>11}  " + "  ".join(f"{'T' + str(n):>11}" for n in range(6)))
for x in (0.1, 0.5, 1.0, 2.0, 4.0):
    row = []
    for n in range(6):
        row.append(sum(coeffs[k] / math.factorial(k) * x ** k for k in range(n + 1)))
    print(f"  {x:>6.1f}  {h(x):>11.6f}  " + "  ".join(f"{v:>11.6f}" for v in row))
print(f"  {'':>6}  {'error':>11}  " + "  ".join(f"{'T' + str(n):>11}" for n in range(6)))
for x in (0.1, 0.5, 1.0, 2.0, 4.0):
    errs = []
    for n in range(6):
        errs.append(abs(h(x) - sum(coeffs[k] / math.factorial(k) * x ** k
                                   for k in range(n + 1))))
    print(f"  {x:>6.1f}  {'':>11}  " + "  ".join(f"{e:>11.2e}" for e in errs))
print("  Two things to read off. Down a column the error falls fast: at x = 0.1 it")
print("  goes from 9.5e-02 to 1.4e-09. Across a ROW it is not monotone -- at x = 2,")
print("  T3's error of 8.3e-01 is worse than T2's 5.1e-01, and at x = 4 the ranking")
print("  is scrambled entirely (T2 1.6, T3 12.3, T4 1.6, T5 6.9). Adding a degree is")
print("  guaranteed to help only in the limit, not at a given x.")
print()
print("  A closed form worth knowing: sin(x) + cos(x) = sqrt(2) sin(x + pi/4), so the")
print("  derivatives at 0 must cycle with period 4 -- which is why the coefficients")
print("  above repeat +1, +1, -1, -1 and you never need to differentiate six times.")
 
print()
print("########## ex_5_5")
print("a. dimensions of df_i/dx")
n = 4
x2 = rng.normal(size=2)
xn = rng.normal(size=n)
yn = rng.normal(size=n)
print(f"  f1: R^2 -> R          df1/dx is 1 x 2")
print(f"  f2: R^n x R^n -> R    df2/dx is 1 x {n}   (and df2/dy is 1 x {n})")
print(f"  f3: R^n -> R^(n x n)  df3/dx is {n} x {n} x {n}   -- a third-order tensor")
print()
print("b. the Jacobians, each checked numerically")
f1 = lambda v: np.sin(v[0]) * np.cos(v[1])
J1 = np.array([[np.cos(x2[0]) * np.cos(x2[1]), -np.sin(x2[0]) * np.sin(x2[1])]])
print(f"  f1: analytic {J1}   numeric {jac(f1, x2)}")
print(f"      gap {np.abs(J1 - jac(f1, x2)).max():.1e}   shape {J1.shape}")
 
f2 = lambda v: np.array([v @ yn])
J2 = yn.reshape(1, -1)
print(f"  f2 wrt x: analytic = y^T, shape {J2.shape}"
      f"   gap {np.abs(J2 - jac(f2, xn)).max():.1e}")
 
# f3(x) = x x^T. The derivative is a tensor: d(f3)_ij / dx_k.
T = np.zeros((n, n, n))
for i in range(n):
    for j in range(n):
        for k in range(n):
            T[i, j, k] = (1.0 if i == k else 0.0) * xn[j] + xn[i] * (1.0 if j == k else 0.0)
# Check it by differencing the matrix-valued function.
Tn = np.zeros((n, n, n))
hh = 1e-6
for k in range(n):
    e = np.zeros(n)
    e[k] = hh
    Tn[:, :, k] = (np.outer(xn + e, xn + e) - np.outer(xn - e, xn - e)) / (2 * hh)
print(f"  f3 = x x^T: d(f3)_ij/dx_k = delta_ik x_j + x_i delta_jk")
print(f"      tensor shape {T.shape}, gap against differences {np.abs(T - Tn).max():.1e}")
print(f"      of {T.size} entries, {int((np.abs(T) > 1e-12).sum())} are nonzero")
 
print()
print("########## ex_5_6")
D, E, F = 4, 3, 5
t = rng.normal(size=D)
ft = lambda v: np.array([np.sin(np.log(v @ v))])
# df/dt = cos(log(t^T t)) * (1/(t^T t)) * 2 t^T
Jt = (np.cos(np.log(t @ t)) / (t @ t) * 2 * t).reshape(1, -1)
print(f"f(t) = sin(log(t^T t)),  t in R^{D}")
print("  df/dt = cos(log(t^T t)) * 2 t^T / (t^T t)      shape 1 x D")
print(f"  analytic {Jt}")
print(f"  numeric  {jac(ft, t)}")
print(f"  gap {np.abs(Jt - jac(ft, t)).max():.1e}")
 
A = rng.normal(size=(D, E))
B = rng.normal(size=(F, D))
X = rng.normal(size=(E, F))
gX = lambda M: np.trace(A @ M.reshape(E, F) @ B)
# dg/dX = (B A)^T
G = (B @ A).T
Gn = np.zeros((E, F))
for i in range(E):
    for j in range(F):
        Ep = np.zeros((E, F))
        Ep[i, j] = 1e-6
        Gn[i, j] = (gX(X + Ep) - gX(X - Ep)) / (2e-6)
print(f"g(X) = tr(A X B),  A {A.shape}, X {X.shape}, B {B.shape}")
print("  dg/dX = (B A)^T                                shape E x F")
print(f"  shape of (BA)^T {G.shape}, gap against differences {np.abs(G - Gn).max():.1e}")
print(f"  and tr(AXB) = tr(BAX) = {np.trace(A @ X @ B):.10f} = {np.trace(B @ A @ X):.10f}")
print("  which is the rearrangement that makes the answer one line.")
 
print()
print("########## ex_5_7")
xd = rng.normal(size=D)
fa = lambda v: np.array([np.log(1 + v @ v)])
Ja = (2 * xd / (1 + xd @ xd)).reshape(1, -1)
print(f"a. f(z) = log(1+z), z = x^T x,  x in R^{D}")
print(f"   df/dz = 1/(1+z)          1 x 1   = {1 / (1 + xd @ xd):.8f}")
print(f"   dz/dx = 2 x^T            1 x {D}")
print(f"   df/dx = 2 x^T / (1 + x^T x)   1 x {D}")
print(f"   gap against differences {np.abs(Ja - jac(fa, xd)).max():.1e}")
 
Am = rng.normal(size=(E, D))
bv = rng.normal(size=E)
fb = lambda v: np.sin(Am @ v + bv)
Jb = np.diag(np.cos(Am @ xd + bv)) @ Am
print(f"b. f(z) = sin(z) elementwise, z = A x + b,  A {Am.shape}")
print(f"   df/dz = diag(cos z)      {E} x {E}")
print(f"   dz/dx = A                {E} x {D}")
print(f"   df/dx = diag(cos(Ax+b)) A    {Jb.shape}")
print(f"   gap against differences {np.abs(Jb - jac(fb, xd)).max():.1e}")
print(f"   the Jacobian is dense ({int((np.abs(Jb) > 1e-12).sum())} of {Jb.size} nonzero)")
print("   but diag(cos z) is a SCALING of A's rows, so never build the E x E matrix.")
 
print()
print("########## ex_5_8")
# a. f(z) = exp(-z/2), z = y^T S^-1 y, y = x - mu
Ssym = rng.normal(size=(D, D))
Ssym = Ssym @ Ssym.T + D * np.eye(D)              # symmetric positive definite
mu_v = rng.normal(size=D)
xv = rng.normal(size=D)
Sinv = np.linalg.inv(Ssym)
f8a = lambda v: np.array([np.exp(-0.5 * (v - mu_v) @ Sinv @ (v - mu_v))])
y = xv - mu_v
z = y @ Sinv @ y
# General S: dz/dy = y^T (S^-1 + S^-T). Symmetric S: 2 y^T S^-1.
J_general = (-0.5 * np.exp(-0.5 * z) * (y @ (Sinv + Sinv.T))).reshape(1, -1)
J_sym = (-np.exp(-0.5 * z) * (y @ Sinv)).reshape(1, -1)
print("a. f(z) = exp(-z/2),  z = y^T S^-1 y,  y = x - mu")
print(f"   df/dz = -1/2 exp(-z/2)             1 x 1")
print(f"   dz/dy = y^T (S^-1 + S^-T)          1 x {D}     -> 2 y^T S^-1 if S is symmetric")
print(f"   dy/dx = I                          {D} x {D}")
print(f"   df/dx = -exp(-z/2) y^T S^-1        1 x {D}")
print(f"   gap against differences: general form {np.abs(J_general - jac(f8a, xv)).max():.1e},"
      f" symmetric form {np.abs(J_sym - jac(f8a, xv)).max():.1e}")
# What happens if S is NOT symmetric and you use the shortcut anyway.
Sns = Ssym + rng.normal(size=(D, D)) * 3.0
Sni = np.linalg.inv(Sns)
f8ns = lambda v: np.array([np.exp(-0.5 * (v - mu_v) @ Sni @ (v - mu_v))])
zn = y @ Sni @ y
Jg = (-0.5 * np.exp(-0.5 * zn) * (y @ (Sni + Sni.T))).reshape(1, -1)
Js = (-np.exp(-0.5 * zn) * (y @ Sni)).reshape(1, -1)
truth = jac(f8ns, xv)
print(f"   with a NON-symmetric S: general form is right to {np.abs(Jg - truth).max():.1e},")
print(f"   the symmetric shortcut is off by {np.abs(Js - truth).max():.3e}"
      f"  ({np.abs(Js - truth).max() / np.abs(truth).max():.1%} relative)")
 
# b. f(x) = tr(x x^T + sigma^2 I)
sig2 = 0.75
f8b = lambda v: np.array([np.trace(np.outer(v, v) + sig2 * np.eye(D))])
J8b = (2 * xd).reshape(1, -1)
print(f"b. f(x) = tr(x x^T + sigma^2 I),  x in R^{D}")
print(f"   the outer product's diagonal is x_i^2, so tr = x^T x + D sigma^2")
print(f"   f(x) = {f8b(xd)[0]:.8f}   and x^T x + D sigma^2 = {xd @ xd + D * sig2:.8f}")
print(f"   df/dx = 2 x^T             1 x {D}")
print(f"   gap against differences {np.abs(J8b - jac(f8b, xd)).max():.1e}")
print(f"   note sigma^2 contributes {D * sig2:.2f} to the VALUE and nothing to the derivative.")
 
# c. f = tanh(z), z = A x + b
M, N = 3, 5
Ac = rng.normal(size=(M, N))
bc = rng.normal(size=M)
xc = rng.normal(size=N)
f8c = lambda v: np.tanh(Ac @ v + bc)
J8c = np.diag(1 - np.tanh(Ac @ xc + bc) ** 2) @ Ac
print(f"c. f = tanh(z) in R^{M},  z = A x + b,  x in R^{N}, A {Ac.shape}")
print(f"   df/dz = diag(1 - tanh^2 z)         {M} x {M}")
print(f"   dz/dx = A                          {M} x {N}")
print(f"   df/dx = diag(1 - tanh^2 z) A       {J8c.shape}")
print(f"   gap against differences {np.abs(J8c - jac(f8c, xc)).max():.1e}")
print(f"   every diagonal entry is in (0, 1]: max here {float((1 - np.tanh(Ac @ xc + bc) ** 2).max()):.6f}")
 
print()
print("########## ex_5_9")
# g(z, nu) = log p(x, z) - log q(z, nu),  z = t(eps, nu)
# The point: nu reaches g by TWO paths. Concrete differentiable choices below.
eps = 0.6
xdat = 1.1
 
 
def logp(z):
    return -0.5 * (xdat - z) ** 2 - 0.5 * z ** 2
 
 
def logq(z, nu):
    m, s = nu[0], np.exp(nu[1])
    return -0.5 * ((z - m) / s) ** 2 - np.log(s) - 0.5 * np.log(2 * np.pi)
 
 
def tfun(e, nu):
    return nu[0] + np.exp(nu[1]) * e                # the reparameterisation
 
 
def G(nu):
    z = tfun(eps, nu)
    return logp(z) - logq(z, nu)
 
 
nu0 = np.array([0.3, -0.4])
print("g(z, nu) = log p(x, z) - log q(z, nu),   z = t(eps, nu)")
print("  dg/dnu = [dlogp/dz - dlogq/dz] dt/dnu   -   dlogq/dnu")
print("           ^ the path THROUGH z                ^ the direct path")
hh = 1e-6
grad_num = np.array([(G(nu0 + np.eye(2)[i] * hh) - G(nu0 - np.eye(2)[i] * hh)) / (2 * hh)
                     for i in range(2)])
z0 = tfun(eps, nu0)
dlogp_dz = -(z0 - xdat) - z0
m, s = nu0[0], np.exp(nu0[1])
dlogq_dz = -(z0 - m) / s ** 2
dt_dnu = np.array([1.0, np.exp(nu0[1]) * eps])
dlogq_dnu = np.array([(z0 - m) / s ** 2, ((z0 - m) ** 2 / s ** 2) - 1.0])
grad_full = (dlogp_dz - dlogq_dz) * dt_dnu - dlogq_dnu
grad_through_z_only = (dlogp_dz - dlogq_dz) * dt_dnu
print(f"  numeric total derivative      {grad_num}")
print(f"  chain rule, both paths        {grad_full}")
print(f"  gap                           {np.abs(grad_full - grad_num).max():.1e}")
print(f"  keeping ONLY the path through z {grad_through_z_only}")
print(f"  that is off by                {np.abs(grad_through_z_only - grad_num).max():.6f}"
      f"   ({np.abs(grad_through_z_only - grad_num).max() / max(np.abs(grad_num).max(), 1e-12):.1%} relative)")
print("  nu appears in g twice -- inside z = t(eps, nu) and again as an argument of")
print("  log q -- so a total derivative needs both terms. This is the reparameterisation")
print("  gradient used to train variational autoencoders, and dropping the direct")
print("  path is the classic implementation bug.")

output

########## ex_5_1
f(x) = log(x^4) sin(x^3)
f'(x) = 4 sin(x^3)/x + 12 x^2 log(x) cos(x^3)
       x        analytic    central diff        gap
    0.40     -1.11611325     -1.11611325    2.2e-11
    0.80      0.95562567      0.95562567    1.2e-12
    1.00      3.36588394      3.36588394    5.7e-11
    1.50    -11.26748685    -11.26748685    1.1e-09
    2.00     -2.86222454     -2.86222454    5.2e-10
  note log(x^4) = 4 log|x| BEFORE differentiating: that is where the 4 and
  the 12 = 4*3 come from, and it is much less error-prone than the chain rule
  on the fourth power.
 
########## ex_5_2
f(x) = 1/(1 + exp(-x))
f'(x) = exp(-x)/(1+exp(-x))^2 = f(x)(1 - f(x))
       x          f(1-f)        exp form    central diff        gap
    -3.0    0.0451766597    0.0451766597    0.0451766597    8.6e-12
    -1.0    0.1966119332    0.1966119332    0.1966119332    7.3e-12
     0.0    0.2500000000    0.2500000000    0.2500000000    3.5e-11
     1.0    0.1966119332    0.1966119332    0.1966119332    6.3e-11
     3.0    0.0451766597    0.0451766597    0.0451766597    4.0e-11
  the maximum of f' is at x = 0, where it equals 0.2500 = 1/4 exactly.
  That 1/4 is the per-layer ceiling behind vanishing gradients in sigmoid nets.
 
########## ex_5_3
f(x) = exp(-(x-mu)^2 / (2 sigma^2))   with mu = 1.3, sigma = 0.7
f'(x) = -(x - mu)/sigma^2 * f(x)
       x        analytic    central diff        gap
   -0.50    0.1346621949    0.1346621949    1.5e-11
    0.60    0.8664723710    0.8664723710    1.4e-12
    1.30   -0.0000000000    0.0000000000    0.0e+00
    2.00   -0.8664723710   -0.8664723710    1.4e-12
    3.10   -0.1346621949   -0.1346621950    2.7e-11
  f'(mu) = -0.0e+00 exactly: the only stationary point is the peak.
 
########## ex_5_4
f(x) = sin(x) + cos(x) at x0 = 0
  derivatives at 0 cycle with period 4:  1, 1, -1, -1, 1, 1, ...
  T0(x) = +1 x^0
  T1(x) = +1 x^0 + +1 x^1
  T2(x) = +1 x^0 + +1 x^1 + -0.5 x^2
  T3(x) = +1 x^0 + +1 x^1 + -0.5 x^2 + -0.166667 x^3
  T4(x) = +1 x^0 + +1 x^1 + -0.5 x^2 + -0.166667 x^3 + +0.0416667 x^4
  T5(x) = +1 x^0 + +1 x^1 + -0.5 x^2 + -0.166667 x^3 + +0.0416667 x^4 + +0.00833333 x^5
 
       x         f(x)           T0           T1           T2           T3           T4           T5
     0.1     1.094838     1.000000     1.100000     1.095000     1.094833     1.094838     1.094838
     0.5     1.357008     1.000000     1.500000     1.375000     1.354167     1.356771     1.357031
     1.0     1.381773     1.000000     2.000000     1.500000     1.333333     1.375000     1.383333
     2.0     0.493151     1.000000     3.000000     1.000000    -0.333333     0.333333     0.600000
     4.0    -1.410446     1.000000     5.000000    -3.000000   -13.666667    -3.000000     5.533333
                error           T0           T1           T2           T3           T4           T5
     0.1                  9.48e-02     5.16e-03     1.62e-04     4.25e-06     8.19e-08     1.41e-09
     0.5                  3.57e-01     1.43e-01     1.80e-02     2.84e-03     2.37e-04     2.31e-05
     1.0                  3.82e-01     6.18e-01     1.18e-01     4.84e-02     6.77e-03     1.56e-03
     2.0                  5.07e-01     2.51e+00     5.07e-01     8.26e-01     1.60e-01     1.07e-01
     4.0                  2.41e+00     6.41e+00     1.59e+00     1.23e+01     1.59e+00     6.94e+00
  Two things to read off. Down a column the error falls fast: at x = 0.1 it
  goes from 9.5e-02 to 1.4e-09. Across a ROW it is not monotone -- at x = 2,
  T3's error of 8.3e-01 is worse than T2's 5.1e-01, and at x = 4 the ranking
  is scrambled entirely (T2 1.6, T3 12.3, T4 1.6, T5 6.9). Adding a degree is
  guaranteed to help only in the limit, not at a given x.
 
  A closed form worth knowing: sin(x) + cos(x) = sqrt(2) sin(x + pi/4), so the
  derivatives at 0 must cycle with period 4 -- which is why the coefficients
  above repeat +1, +1, -1, -1 and you never need to differentiate six times.
 
########## ex_5_5
a. dimensions of df_i/dx
  f1: R^2 -> R          df1/dx is 1 x 2
  f2: R^n x R^n -> R    df2/dx is 1 x 4   (and df2/dy is 1 x 4)
  f3: R^n -> R^(n x n)  df3/dx is 4 x 4 x 4   -- a third-order tensor
 
b. the Jacobians, each checked numerically
  f1: analytic [[ 0.169624 -0.696987]]   numeric [[ 0.169624 -0.696987]]
      gap 6.0e-11   shape (1, 2)
  f2 wrt x: analytic = y^T, shape (1, 4)   gap 7.2e-11
  f3 = x x^T: d(f3)_ij/dx_k = delta_ik x_j + x_i delta_jk
      tensor shape (4, 4, 4), gap against differences 1.8e-10
      of 64 entries, 28 are nonzero
 
########## ex_5_6
f(t) = sin(log(t^T t)),  t in R^4
  df/dt = cos(log(t^T t)) * 2 t^T / (t^T t)      shape 1 x D
  analytic [[-0.005728  0.0259    0.020124 -0.033601]]
  numeric  [[-0.005728  0.0259    0.020124 -0.033601]]
  gap 2.6e-11
g(X) = tr(A X B),  A (4, 3), X (3, 5), B (5, 4)
  dg/dX = (B A)^T                                shape E x F
  shape of (BA)^T (3, 5), gap against differences 8.9e-10
  and tr(AXB) = tr(BAX) = -6.1094555522 = -6.1094555522
  which is the rearrangement that makes the answer one line.
 
########## ex_5_7
a. f(z) = log(1+z), z = x^T x,  x in R^4
   df/dz = 1/(1+z)          1 x 1   = 0.25620140
   dz/dx = 2 x^T            1 x 4
   df/dx = 2 x^T / (1 + x^T x)   1 x 4
   gap against differences 5.5e-11
b. f(z) = sin(z) elementwise, z = A x + b,  A (3, 4)
   df/dz = diag(cos z)      3 x 3
   dz/dx = A                3 x 4
   df/dx = diag(cos(Ax+b)) A    (3, 4)
   gap against differences 1.1e-10
   the Jacobian is dense (12 of 12 nonzero)
   but diag(cos z) is a SCALING of A's rows, so never build the E x E matrix.
 
########## ex_5_8
a. f(z) = exp(-z/2),  z = y^T S^-1 y,  y = x - mu
   df/dz = -1/2 exp(-z/2)             1 x 1
   dz/dy = y^T (S^-1 + S^-T)          1 x 4     -> 2 y^T S^-1 if S is symmetric
   dy/dx = I                          4 x 4
   df/dx = -exp(-z/2) y^T S^-1        1 x 4
   gap against differences: general form 4.9e-11, symmetric form 4.9e-11
   with a NON-symmetric S: general form is right to 2.7e-11,
   the symmetric shortcut is off by 1.411e-01  (62.5% relative)
b. f(x) = tr(x x^T + sigma^2 I),  x in R^4
   the outer product's diagonal is x_i^2, so tr = x^T x + D sigma^2
   f(x) = 5.90317933   and x^T x + D sigma^2 = 5.90317933
   df/dx = 2 x^T             1 x 4
   gap against differences 4.8e-10
   note sigma^2 contributes 3.00 to the VALUE and nothing to the derivative.
c. f = tanh(z) in R^3,  z = A x + b,  x in R^5, A (3, 5)
   df/dz = diag(1 - tanh^2 z)         3 x 3
   dz/dx = A                          3 x 5
   df/dx = diag(1 - tanh^2 z) A       (3, 5)
   gap against differences 7.8e-11
   every diagonal entry is in (0, 1]: max here 0.692457
 
########## ex_5_9
g(z, nu) = log p(x, z) - log q(z, nu),   z = t(eps, nu)
  dg/dnu = [dlogp/dz - dlogq/dz] dt/dnu   -   dlogq/dnu
           ^ the path THROUGH z                ^ the direct path
  numeric total derivative      [-0.304384  0.877579]
  chain rule, both paths        [-0.304384  0.877579]
  gap                           1.4e-10
  keeping ONLY the path through z [0.590711 0.237579]
  that is off by                0.895095   (102.0% relative)
  nu appears in g twice -- inside z = t(eps, nu) and again as an argument of
  log q -- so a total derivative needs both terms. This is the reparameterisation
  gradient used to train variational autoencoders, and dropping the direct
  path is the classic implementation bug.

pch.quizTag Check your understanding

1. In 5.1, what is the single simplification that makes the answer short?

   A Using the product rule before the chain rule B Rewriting log(x^4) as 4 log|x| BEFORE differentiating — that turns a chain rule on a fourth power into a constant times a logarithm, and it is where both the 4 and the 12 come from C Substituting u = x^3 D Expanding sin(x^3) as a series
   pch.quizShowAnswer

   B — Rewriting log(x^4) as 4 log|x| BEFORE differentiating — that turns a chain rule on a fourth power into a constant times a logarithm, and it is where both the 4 and the 12 come from — Simplify, then differentiate. The same instinct applies to 5.6's trace (rearrange cyclically first) and 5.8b's outer product (write out the diagonal first): in all three the algebra before the calculus is the whole exercise.

2. 5.2's derivative is f(1-f), whose maximum is 1/4. Why does that number matter?

   A It makes the sigmoid easy to differentiate B Every sigmoid layer multiplies the backward signal by at most 1/4, so ten stacked sigmoids attenuate a gradient by at least 4^-10, about 1e-6 — the vanishing gradient problem as a one-line calculation C It is the variance of the sigmoid D It bounds the sigmoid's output
   pch.quizShowAnswer

   B — Every sigmoid layer multiplies the backward signal by at most 1/4, so ten stacked sigmoids attenuate a gradient by at least 4^-10, about 1e-6 — the vanishing gradient problem as a one-line calculation — This is why ReLU replaced the sigmoid in deep networks: its derivative is exactly 1 wherever the unit is active, so the product of layer factors does not shrink automatically. 5.8c's tanh has the same structure with a ceiling of 1.

3. In 5.5, what is the shape of the derivative of f3(x) = x x-transpose, and why?

   A n by n, because the output is n by n B n by n by n. The rule is output shape followed by input shape: the output is a matrix, so the derivative gains an index and stops being a matrix C 1 by n, because it is a scalar function D n by 1
   pch.quizShowAnswer

   B — n by n by n. The rule is output shape followed by input shape: the output is a matrix, so the derivative gains an index and stops being a matrix — There is no clean matrix identity for this derivative because the result is not a matrix. Writing the entry out — (f3)_ij = x_i x_j, differentiate with respect to x_k, collect Kronecker deltas — gives delta_ik x_j + x_i delta_jk and needs no lookup table.

4. In 5.8a, the derivative of y-transpose M y with respect to y is written y-transpose (M + M-transpose) rather than 2 y-transpose M. When does it matter?

   A Never, they are always equal B Whenever M is not symmetric. Measured on a deliberately non-symmetric S, the shortcut is 62.5 percent wrong — a plausible-looking answer with the right shape. It is safe here only because a covariance matrix is symmetric C Only for singular M D Only in high dimensions
   pch.quizShowAnswer

   B — Whenever M is not symmetric. Measured on a deliberately non-symmetric S, the shortcut is 62.5 percent wrong — a plausible-looking answer with the right shape. It is safe here only because a covariance matrix is symmetric — The two forms agree to 4.9e-11 on a symmetric S and disagree by 1.4e-01 on a non-symmetric one. Knowing WHY the shortcut is licensed is the difference between using an identity and reciting it.

5. In 5.9, what happens if you keep only the path through z?

   A The gradient is slightly too small B It is 102 percent wrong and the first component changes SIGN — measured -0.304384 for the correct gradient against +0.590711 for the truncated one. A gradient pointing the wrong way does not converge slowly, it diverges C Nothing, the direct term is zero D The answer is correct but harder to compute
   pch.quizShowAnswer

   B — It is 102 percent wrong and the first component changes SIGN — measured -0.304384 for the correct gradient against +0.590711 for the truncated one. A gradient pointing the wrong way does not converge slowly, it diverges — The tempting argument — 'the expectation of d log q / d nu is zero by the score-function identity' — is true about the expectation and false about a single sample, and a single sample is what you differentiate. This is the classic implementation bug in variational autoencoders.

🧪 Try It Yourself

Exercise 1 – 5.1, 5.2 and 5.3

pch.exerciseTag pch.exerciseTitle

Exercise 2 – 5.4, and the row that is not monotone

pch.exerciseTag pch.exerciseTitle

Exercise 3 – 5.5’s tensor derivative

pch.exerciseTag pch.exerciseTitle

Exercise 4 – 5.6’s trace, in one line

pch.exerciseTag pch.exerciseTitle

Exercise 5 – 5.9, with and without the direct path

pch.exerciseTag pch.exerciseTitle

Recall card

• Simplify before differentiating. 5.1’s log(x^4) = 4 log|x|, 5.6’s cyclic trace rearrangement, and 5.8b’s outer-product diagonal are each one algebraic step that removes most of the calculus.
• 5.2: the sigmoid’s derivative is f(1-f), a function of the OUTPUT — which is why a sigmoid backward pass reuses the stored activation instead of recomputing an exponential.
• The number 1/4. f(1-f) peaks at exactly 0.25, so ten stacked sigmoids attenuate a gradient by at least 4^-10, about 1e-6. That is the vanishing-gradient problem in one line, and why ReLU replaced the sigmoid.
• 5.3: the Gaussian’s derivative is a multiple of itself, -(x-mu)/sigma^2 times f, so its only stationary point is the peak and it is antisymmetric about mu.
• 5.4: use the period-4 cycle, not six differentiations. sin x + cos x = sqrt(2) sin(x + pi/4), so the derivatives at 0 repeat +1, +1, -1, -1.
• 5.4’s error table is not monotone across a row. At x = 2, T3’s error of 8.3e-01 is worse than T2’s 5.1e-01; at x = 4 the ranking is scrambled. Down a column at x = 0.1 it falls from 9.5e-02 to 1.4e-09.
• The derivative’s shape is the output’s shape followed by the input’s. So a matrix-valued function of a vector has a third-order derivative: for x x-transpose it is n by n by n, with entries delta_ik x_j + x_i delta_jk.
• When there is no matrix identity, drop to indices. Write the (i,j) entry, differentiate with respect to x_k, collect Kronecker deltas. It always works and never needs a lookup table.
• 5.6: tr(AXB) = tr(BAX), so dg/dX = (BA)-transpose, shape E by F, which matches X — the free shape check on every matrix-calculus answer.
• An elementwise function has a DIAGONAL Jacobian. 5.7b and 5.8c both come out as diag(something) times A, which in code is a broadcast, never an E-by-E allocation.
• 5.8a: d(y-transpose M y)/dy is y-transpose (M + M-transpose), and only 2 y-transpose M when M is symmetric. Measured on a non-symmetric S the shortcut is 62.5 percent wrong.
• 5.8b: there was never a matrix. tr(x x-transpose + sigma^2 I) = x-transpose x + D sigma^2, so the derivative is 2 x-transpose and sigma^2 contributes nothing to it.
• 5.8c is one network layer, and every diagonal entry of 1 - tanh^2(z) lies in (0, 1] — the factors whose product Equation 5.118 takes K times.
• 5.9: nu reaches g by TWO paths, inside z = t(eps, nu) and directly as an argument of log q, so the total derivative needs both terms.
• Keeping only 5.9’s z path is 102 percent wrong and flips a sign — measured -0.304384 against +0.590711. The ‘the score term has zero expectation’ argument is about an expectation, and you are differentiating one sample.

Next: Chapter 5 Formula Sheet — every definition, theorem and identity in the chapter on one page.