Chapter 5 Formula Sheet

Reference, not teaching. Each entry names the book’s number, states the result, gives the shape of every object, and names the condition it needs. Read the concept pages first; come back here when you are working.

The one-line summary

The whole chapter is one rule applied at four levels of generality:

$$\underbrace{f'(x)}_{\text{scalar}} \;\subset\; \underbrace{\nabla f \in \mathbb{R}^{1\times n}}_{\text{scalar of a vector}} \;\subset\; \underbrace{\mathbf{J} \in \mathbb{R}^{m\times n}}_{\text{vector of a vector}} \;\subset\; \underbrace{\text{tensors}}_{\text{matrix of a matrix}}$$

and at every level the chain rule is the same matrix product, read left to right, with the shapes lining up. That is the payoff of §5.2’s decision to make the gradient a row vector.

The shape rule, which subsumes most of the chapter

The derivative’s shape is the output’s shape followed by the input’s shape.

$f$ maps	$\mathrm{d}f/\mathrm{d}x$ shape	Name
$\mathbb{R} \to \mathbb{R}$	scalar	derivative
$\mathbb{R}^n \to \mathbb{R}$	$1 \times n$	gradient (Eq 5.40)
$\mathbb{R}^n \to \mathbb{R}^m$	$m \times n$	Jacobian (Eq 5.56)
$\mathbb{R}^n \to \mathbb{R}$, twice	$n \times n$	Hessian (Eq 5.147)
$\mathbb{R}^{n} \to \mathbb{R}^{m}$, twice	$m \times n \times n$	a tensor (§5.7 Remark)
$\mathbb{R}^{p\times q} \to \mathbb{R}^{m\times n}$	$(m\times n)\times(p\times q)$	a 4-tensor (§5.4)

If you can state the shape before computing anything, the rest is bookkeeping. Every dimension error in this chapter is a violation of that one line.

§5.1 Differentiation of univariate functions

#	Result	Condition
Def 5.1	difference quotient $\dfrac{\delta y}{\delta x} = \dfrac{f(x+\delta x) - f(x)}{\delta x}$	—
Def 5.2	derivative $f'(x) = \lim\limits_{h\to0}\dfrac{f(x+h)-f(x)}{h}$	limit must exist
—	product: $(uv)' = u'v + uv'$	both differentiable
—	quotient: $\left(\dfrac{u}{v}\right)' = \dfrac{u'v - uv'}{v^2}$	$v \neq 0$
—	sum: $(u+v)' = u' + v'$	both differentiable
Eq 5.32	chain rule: $\bigl(g(f(x))\bigr)' = g'(f(x))f'(x)$	inner differentiable at $x$, outer at $f(x)$
Def 5.3	Taylor polynomial $T_n(x) = \sum_{k=0}^{n}\dfrac{f^{(k)}(x_0)}{k!}(x-x_0)^k$	$f$ $n$ times differentiable at $x_0$
Def 5.4	Taylor series: the same sum to $\infty$	$f \in C^\infty$; convergence is a separate question

The difference quotient has a floor in floating point

Def 5.2’s limit does not exist numerically. Measured on Example 5.14’s function: the central difference improves down to about $h = 10^{-6}$, reaching $9.8 \times 10^{-12}$, and then gets worse — subtracting two nearly equal numbers destroys digits. A second difference divides by $h^2$ and hits its floor about a thousand times earlier. This is why automatic differentiation, which has no $h$ at all, reaches $2.2\times10^{-16}$.

§5.2 Partial differentiation and gradients

#	Result	Shape
Def 5.5	$\dfrac{\partial f}{\partial x_i} = \lim\limits_{h\to0}\dfrac{f(x_1,\dots,x_i+h,\dots,x_n) - f(\mathbf{x})}{h}$	scalar
Eq 5.40	$\nabla_\mathbf{x}f = \dfrac{\mathrm{d}f}{\mathrm{d}\mathbf{x}} = \begin{bmatrix}\dfrac{\partial f}{\partial x_1} & \cdots & \dfrac{\partial f}{\partial x_n}\end{bmatrix}$	$1 \times n$, a row
Eq 5.46	product rule: $\dfrac{\partial}{\partial\mathbf{x}}\bigl(f(\mathbf{x})g(\mathbf{x})\bigr) = \dfrac{\partial f}{\partial\mathbf{x}}g + f\dfrac{\partial g}{\partial\mathbf{x}}$	$1 \times n$
Eq 5.47	sum rule	$1 \times n$
Eq 5.48	chain rule: $\dfrac{\partial}{\partial\mathbf{x}}(g\circ f) = \dfrac{\partial g}{\partial f}\dfrac{\partial f}{\partial\mathbf{x}}$	$1 \times n$
Eq 5.49	$\dfrac{\mathrm{d}f}{\mathrm{d}t} = \dfrac{\partial f}{\partial x_1}\dfrac{\partial x_1}{\partial t} + \dfrac{\partial f}{\partial x_2}\dfrac{\partial x_2}{\partial t}$	scalar
Eq 5.50	$\dfrac{\mathrm{d}f}{\mathrm{d}(s,t)} = \dfrac{\partial f}{\partial\mathbf{x}}\dfrac{\partial\mathbf{x}}{\partial(s,t)}$	$1 \times 2$

Two facts about the gradient, both measurable. It points in the direction of steepest ascent, and the steepest rate equals its norm — swept over 360 directions, nothing beats $\lVert\nabla f\rVert$ and the exact gradient direction attains it to $10^{-12}$. And it is perpendicular to the level set: measured across a whole field, the worst normalised deviation is $10^{-12}$.

§5.3 Gradients of vector-valued functions

#	Result	Shape
Eq 5.55	$\dfrac{\partial \mathbf{f}}{\partial x_i} \in \mathbb{R}^{m}$, a column	$m \times 1$
Eq 5.56	Jacobian $\mathbf{J} = \nabla_\mathbf{x}\mathbf{f} = \begin{bmatrix}\dfrac{\partial\mathbf{f}}{\partial x_1} & \cdots & \dfrac{\partial\mathbf{f}}{\partial x_n}\end{bmatrix}$	$m \times n$
Eq 5.58	$J_{ij} = \dfrac{\partial f_i}{\partial x_j}$	—
Def 5.6	$\det\mathbf{J}$ is the local volume scale factor	$n = m$
Eq 5.68	chain rule for a composition, as a matrix product	$(m\times k)(k\times n)$

Linear maps have constant Jacobians. For $\mathbf{f}(\mathbf{x}) = \mathbf{A}\mathbf{x}$, $\mathbf{J} = \mathbf{A}$ everywhere. For a nonlinear map the Jacobian is only local — measured on a fold-free patch, the ratio of a small square’s image area to $\lvert\det\mathbf{J}\rvert$ converges to $1$ at first order in the patch size.

Where det J changes sign, area is not what you measured

A shoelace area is signed. If the patch straddles the curve where $\det\mathbf{J} = 0$ the polygon folds over itself and the formula returns the difference of two signed pieces, not their sum. Measured on a map with $\det\mathbf{J} = 1 - 1.2uv$: a square at $(0.8, 0.6)$ straddles the fold at $uv = 0.8333$ and its “area” came out $0.04$ against a $\lvert\det\mathbf{J}\rvert$ of $0.424$. Rebased away from the fold, the agreement is clean.

§5.4 Gradients of matrices

#	Result	Shape
—	$\dfrac{\mathrm{d}\mathbf{f}}{\mathrm{d}\mathbf{A}}$ for $\mathbf{f}\in\mathbb{R}^m$, $\mathbf{A}\in\mathbb{R}^{p\times q}$	$m \times (p \times q)$
Eq 5.85	$\mathbf{f} = \mathbf{A}\mathbf{x}$: $\dfrac{\partial f_i}{\partial A_{i,:}} = \mathbf{x}^\top$, and $\mathbf{0}$ for every other row	$1 \times n$ per slice
—	$\dfrac{\mathrm{d}\mathbf{K}}{\mathrm{d}\mathbf{R}}$ for $\mathbf{K} = \mathbf{R}^\top\mathbf{R}$	$(n\times n)\times(m\times n)$

Two ways to organise the same numbers. Either keep the tensor shape, or flatten the matrix to a $pq$-vector and get an ordinary Jacobian. The book does both; the flattened form is what a library computes, and the tensor form is what makes the shapes comprehensible.

§5.5 Useful identities

With the book’s Equations 5.99 to 5.108. Every one of these is verified numerically on the identities page.

#	Identity
Eq 5.99	$\dfrac{\partial}{\partial\mathbf{X}}\mathbf{f}(\mathbf{X})^\top = \left(\dfrac{\partial \mathbf{f}(\mathbf{X})}{\partial\mathbf{X}}\right)^\top$
Eq 5.100	$\dfrac{\partial}{\partial\mathbf{X}}\mathrm{tr}\bigl(\mathbf{f}(\mathbf{X})\bigr) = \mathrm{tr}\left(\dfrac{\partial \mathbf{f}(\mathbf{X})}{\partial\mathbf{X}}\right)$
Eq 5.101	$\dfrac{\partial}{\partial\mathbf{X}}\det\bigl(\mathbf{f}(\mathbf{X})\bigr) = \det\bigl(\mathbf{f}(\mathbf{X})\bigr)\mathrm{tr}\left(\mathbf{f}(\mathbf{X})^{-1}\dfrac{\partial \mathbf{f}(\mathbf{X})}{\partial\mathbf{X}}\right)$
Eq 5.102	$\dfrac{\partial}{\partial\mathbf{X}}\mathbf{f}(\mathbf{X})^{-1} = -\mathbf{f}(\mathbf{X})^{-1}\dfrac{\partial \mathbf{f}(\mathbf{X})}{\partial\mathbf{X}}\mathbf{f}(\mathbf{X})^{-1}$
Eq 5.103	$\dfrac{\partial \mathbf{a}^\top\mathbf{X}^{-1}\mathbf{b}}{\partial\mathbf{X}} = -(\mathbf{X}^{-1})^\top\mathbf{a}\mathbf{b}^\top(\mathbf{X}^{-1})^\top$
Eq 5.104	$\dfrac{\partial \mathbf{x}^\top\mathbf{a}}{\partial\mathbf{x}} = \mathbf{a}^\top$
Eq 5.105	$\dfrac{\partial \mathbf{a}^\top\mathbf{X}\mathbf{b}}{\partial\mathbf{X}} = \mathbf{a}\mathbf{b}^\top$
Eq 5.106	$\dfrac{\partial \mathbf{x}^\top\mathbf{B}\mathbf{x}}{\partial\mathbf{x}} = \mathbf{x}^\top(\mathbf{B} + \mathbf{B}^\top)$
Eq 5.107	$\dfrac{\partial}{\partial\mathbf{s}}(\mathbf{x} - \mathbf{A}\mathbf{s})^\top\mathbf{W}(\mathbf{x}-\mathbf{A}\mathbf{s}) = -2(\mathbf{x}-\mathbf{A}\mathbf{s})^\top\mathbf{W}\mathbf{A}$
Eq 5.108	the same for symmetric $\mathbf{W}$

Equation 5.106 is the one people get wrong

The derivative of a quadratic form is $\mathbf{x}^\top(\mathbf{B} + \mathbf{B}^\top)$, not $2\mathbf{x}^\top\mathbf{B}$. They agree only when $\mathbf{B}$ is symmetric. Measured on a deliberately non-symmetric matrix, the shortcut is 62.5% relative wrong — a plausible answer with the right shape. Equation 5.108’s symmetric shortcut is worse still if you apply it unconditionally: it came out 1.20 relative (120%) in one test and $2.9\times10^{-1}$ in another.

§5.6 Backpropagation and automatic differentiation

#	Result
Eq 5.109	the example function $f(x) = \sqrt{x^2+\exp(x^2)} + \cos(x^2+\exp(x^2))$
Eq 5.110	its explicit derivative — 11 operations, and it recomputes $x^2$ and $\exp(x^2)$
Eq 5.111	$\mathbf{f}_0 = \mathbf{x}$, $\mathbf{f}_i = \sigma_i(\mathbf{A}_{i-1}\mathbf{f}_{i-1} + \mathbf{b}_{i-1})$
Eq 5.115–5.118	the cascade: $\dfrac{\partial L}{\partial\boldsymbol{\theta}_i}$ extends the previous product by one factor
Eq 5.119–5.121	forward and reverse mode as two bracketings of one product of Jacobians
Eq 5.123–5.128	Example 5.14’s forward pass, six intermediates
Eq 5.129–5.134	its elementary local derivatives
Eq 5.135–5.142	its reverse sweep
Eq 5.143–5.145	the general algorithm; Eq 5.145’s sum is over all children of a node

The costs, measured. Reverse mode costs one sweep per output, forward mode one per input. On Example 5.14: 6 forward operations against 16 in the reverse sweep — a factor of $2.67$, a constant rather than a blow-up. With a scalar loss the crossover is at $n = 3$, so for a network with millions of parameters reverse mode is not a preference but the only option. And AD’s error was $2.2\times10^{-16}$ against the best of 200 central differences at $9.8\times10^{-12}$.

A node with two children needs a sum

Equation 5.145 sums over children, and Example 5.14’s node $a$ is the only one with two — so Equation 5.137 is the only adjoint in the whole example that is a sum rather than a single product, and therefore the only one easy to forget. Dropping its second term leaves the gradient the right sign and order of magnitude but 9.5% to 47.8% wrong, worst for small $x$, which inverts the usual debugging instinct.

§5.7 Higher-order derivatives

#	Result	Condition
—	$\dfrac{\partial^2 f}{\partial y\,\partial x} = \dfrac{\partial}{\partial y}\!\left(\dfrac{\partial f}{\partial x}\right)$: the operator nearest $f$ acts first	—
Eq 5.146	$\dfrac{\partial^2 f}{\partial x\,\partial y} = \dfrac{\partial^2 f}{\partial y\,\partial x}$	twice continuously differentiable
Eq 5.147	$H = \begin{bmatrix}\partial^2_{xx}f & \partial^2_{xy}f \\ \partial^2_{xy}f & \partial^2_{yy}f\end{bmatrix}$, symmetric, written $\nabla^2_{x,y}f$	$n \times n$ in general
—	curvature along a unit $\mathbf{d}$ is $\mathbf{d}^\top H\mathbf{d}$, bounded by the eigenvalues and attaining both	$H$ symmetric
Remark	a vector field’s Hessian is an $(m\times n\times n)$ tensor	—

The second-order test has four outcomes. All eigenvalues positive: minimum. All negative: maximum. Mixed: saddle. Some zero: the test says nothing — and $x_1^2+x_2^4$, $x_1^2+x_2^3$ and $x_1^2-x_2^4$ all share the Hessian $\begin{bmatrix}2&0\\0&0\end{bmatrix}$ at the origin while being, respectively, a strict minimum and two non-optima.

Never test an eigenvalue against zero

A central second difference has a truncation floor of exactly $2h^2$, and it is positive — measured from $2.0\times10^{-4}$ at $h = 10^{-2}$ down to $2.0\times10^{-14}$ at $h=10^{-7}$, reporting a singular Hessian as positive definite at every step size. Compare against a tolerance scaled to the entries.

A symmetric stencil cannot verify Equation 5.146

The five-point mixed stencil computes both orders from the same four corner values, so its gap is 0.0e+00 for every function — including the counterexample $xy(x^2-y^2)/(x^2+y^2)$, whose two iterated limits at the origin are $-1$ and $+1$, a gap of exactly $2$ at every step size.

§5.8 Linearization and multivariate Taylor series

#	Result	Shape
Eq 5.148	$f(\mathbf{x}) \approx f(\mathbf{x}_0) + (\nabla_\mathbf{x}f)(\mathbf{x}_0)(\mathbf{x}-\mathbf{x}_0)$	$(1\times D)(D\times 1)$
Def 5.7 / Eq 5.151	$f(\mathbf{x}) = \sum_{k=0}^{\infty}\dfrac{D^k_\mathbf{x}f(\mathbf{x}_0)}{k!}\boldsymbol{\delta}^k$, $\boldsymbol{\delta} := \mathbf{x}-\mathbf{x}_0$	—
Def 5.8 / Eq 5.152	$T_n(\mathbf{x}) = \sum_{k=0}^{n}\dfrac{D^k_\mathbf{x}f(\mathbf{x}_0)}{k!}\boldsymbol{\delta}^k$	—
Eq 5.153	$\boldsymbol{\delta}^2 = \boldsymbol{\delta}\otimes\boldsymbol{\delta} = \boldsymbol{\delta}\boldsymbol{\delta}^\top$, entries $\delta_i\delta_j$	$D\times D$
Eq 5.154	$\boldsymbol{\delta}^3$, entries $\delta_i\delta_j\delta_k$	$D\times D\times D$
Eq 5.155	$D^k f\,\boldsymbol{\delta}^k = \sum_{i_1}\cdots\sum_{i_k} D^kf[i_1..i_k]\delta_{i_1}\cdots\delta_{i_k}$	scalar
Eq 5.156	$k=0$: $f(\mathbf{x}_0)$	scalar
Eq 5.157	$k=1$: $\nabla_\mathbf{x}f(\mathbf{x}_0)\boldsymbol{\delta}$ — einsum("i,i", Df1, d)	scalar
Eq 5.158	$k=2$: $\mathrm{tr}(H\boldsymbol{\delta}\boldsymbol{\delta}^\top) = \boldsymbol{\delta}^\top H\boldsymbol{\delta}$	scalar
Eq 5.159	$= \sum_i\sum_j H_{ij}\delta_i\delta_j$ — einsum("ij,i,j", Df2, d, d)	scalar
Eq 5.160	$k=3$ — einsum("ijk,i,j,k", Df3, d, d, d)	scalar

The measured facts. The order-$n$ error scales like $r^{n+1}$: fitted slopes $1.067$, $2.005$, $3.019$ against the predicted $1$, $2$, $3$. But that is a statement about $r \to 0$ — on a fixed window a higher-order model can be worse, and at $r = 3.2$ a quadratic model’s error of $7.77$ was five times the constant model’s $1.40$.

The cost. $D^k$ entries at order $k$; $\binom{D+k-1}{k}$ distinct after Equation 5.146’s symmetry. At $D = 10^6$: $5.0\times10^{11}$ at $k = 2$ (4 TB in float32) and $1.7\times10^{17}$ at $k = 3$. Symmetry halves the constant and does nothing to the exponent.

The cross term keeps a factor of 2

Equation 5.159’s double sum counts $(i,j)$ and $(j,i)$ separately. Expanding $\tfrac12\boldsymbol{\delta}^\top H\boldsymbol{\delta}$ for Example 5.15 gives $(x-1)^2 + 2(x-1)(y-2) + 6(y-2)^2$ — a $2$ on the middle term and no $\tfrac12$ anywhere. Wrong by a factor of two on exactly one term is the hardest kind of error to spot.

Numbers worth remembering

Quantity	Value	Where
max of the sigmoid’s derivative	exactly $\tfrac14$, at $x = 0$	Exercise 5.2
reverse-vs-forward crossover, scalar loss	$n = 3$	§5.6
reverse sweep vs forward pass, Example 5.14	$2.67\times$	§5.6
AD error vs best central difference	$2.2\text{e-}16$ vs $9.8\text{e-}12$	§5.6
cost of dropping Eq 5.137’s second term	9.5% to 47.8%, worst for small $x$	§5.6
second-difference truncation floor	exactly $2h^2$, and positive	§5.7
Schwarz counterexample, gap at the origin	exactly $2$, at every step size	§5.7
Newton vs gradient descent, convex test	8 steps vs 426, 3 of 4 rates diverged	§5.7
gradient-descent steps vs $\kappa$	18, 94, 939, 9384 for $\kappa$ = 2, 10, 100, 1000	§5.7
P(random symmetric $H$ positive definite)	$0.498$, $0.145$, $0.0272$, $0.0024$ for $n$ = 1 to 4	§5.7
fitted Taylor error orders	$1.067$, $2.005$, $3.019$	§5.8
Hessian entries at $D = 10^6$	$10^{12}$ stored, $5.0\times10^{11}$ distinct	§5.8
non-symmetric quadratic-form shortcut error	62.5% relative	Eq 5.106
dropping Exercise 5.9’s direct path	102% relative, and a sign flip	Exercise 5.9

The exercises, in one line each

#	Answer
5.1	$\dfrac{4\sin(x^3)}{x} + 12x^2\log\lvert x\rvert\cos(x^3)$ — simplify $\log(x^4) = 4\log\lvert x\rvert$ first
5.2	$f(1-f)$, a function of the output; maximum $\tfrac14$
5.3	$-\dfrac{x-\mu}{\sigma^2}f(x)$ — a multiple of itself
5.4	coefficients cycle $+1, +1, -1, -1$, because $\sin x + \cos x = \sqrt2\sin(x + \pi/4)$
5.5	$1\times2$; $1\times n$; $n\times n\times n$ with entries $\delta_{ik}x_j + x_i\delta_{jk}$
5.6	$\cos(\log\mathbf{t}^\top\mathbf{t})\dfrac{2\mathbf{t}^\top}{\mathbf{t}^\top\mathbf{t}}$; and $(\mathbf{B}\mathbf{A})^\top$ via the cyclic trace
5.7	$\dfrac{2\mathbf{x}^\top}{1+\mathbf{x}^\top\mathbf{x}}$; and $\operatorname{diag}(\cos\mathbf{z})\mathbf{A}$
5.8	$-f\cdot\mathbf{y}^\top\mathbf{S}^{-1}$; $2\mathbf{x}^\top$; $\operatorname{diag}(1-\tanh^2\mathbf{z})\mathbf{A}$
5.9	$\left(\partial_z\log p - \partial_z\log q\right)\partial_\nu t - \partial_\nu\log q$ — two paths to $\boldsymbol{\nu}$

Where each section is used later

Section	Used in
§5.1 chain rule	everywhere; §5.6 is this applied mechanically
§5.2 gradient as a row	Chapter 7’s optimality conditions, Chapter 9’s normal equations
§5.3 Jacobian	Chapter 6’s change of variables (§6.7), normalising flows
§5.4 matrix gradients	Chapter 9’s linear regression, Chapter 10’s PCA
§5.5 identities	anywhere a closed-form gradient beats differentiating
§5.6 backpropagation	every neural network ever trained
§5.7 Hessian	Chapter 7’s Newton and convexity, the Laplace approximation
§5.8 Taylor	Chapter 7’s trust regions, Gauss–Newton, the Laplace approximation

The eight mistakes this chapter is designed to prevent

1. Writing the gradient as a column. Equation 5.40 makes it a row so the chain rule is a left-to-right matrix product. Get this wrong and every subsequent shape needs a transpose you will forget.
2. Using $2\mathbf{x}^\top\mathbf{B}$ for a quadratic form’s derivative. It is $\mathbf{x}^\top(\mathbf{B}+\mathbf{B}^\top)$ unless $\mathbf{B}$ is symmetric. Measured cost of the shortcut on a non-symmetric matrix: 62.5%.
3. Forgetting Equation 5.145’s sum at a node with more than one child. Costs 9.5% to 47.8% on Example 5.14, and looks plausible.
4. Testing a Hessian eigenvalue against zero. The differencing floor is $+2h^2$, so a singular Hessian reads as positive definite at every $h$.
5. Treating a zero eigenvalue as “no optimum”. It means ask a higher derivative: three functions share one singular Hessian and have three different answers.
6. Expecting Newton’s method to find a minimum. It solves $\nabla f = \mathbf{0}$, with no preference. On $x_1^2 - x_2^2$ one step lands exactly on the saddle and reports success.
7. Reading “higher order is better” as a statement about your window. It is a statement about $r \to 0$; at $r = 3.2$ the quadratic model was five times worse than the constant one.
8. Dropping the second path in a total derivative when a variable appears more than once. Exercise 5.9’s truncated gradient is 102% wrong and points the opposite way.

Every NumPy call in the chapter

Task	Call
numerical gradient	(f(x+h) - f(x-h)) / (2*h), per component
Jacobian, column by column	same, stacked with np.column_stack
local volume factor	np.linalg.det(J)
contract Eq 5.157	np.einsum("i,i", Df1, d)
contract Eq 5.159	np.einsum("ij,i,j", Df2, d, d)
contract Eq 5.160	np.einsum("ijk,i,j,k", Df3, d, d, d)
quadratic form, cheaply	d @ H @ d, never np.trace(H @ np.outer(d, d))
Hessian eigenvalues	np.linalg.eigvalsh(H) — symmetric only, and it does not check
is $H$ positive definite	np.linalg.cholesky(H) in a try, about half the cost
Newton step	x - np.linalg.solve(H, grad), never inv(H) @ grad
elementwise Jacobian times a matrix	np.cos(z)[:, None] * A, never np.diag(...) @ A

pch.quizTag Check your understanding

1. What single rule determines the shape of every derivative in this chapter?

   A The derivative is always a matrix B The derivative's shape is the output's shape followed by the input's shape — so a scalar of a vector is 1 by n, a vector of a vector is m by n, and a matrix of a vector is m by n by n C The derivative has the same shape as the input D The derivative's shape depends on the chain rule used
   pch.quizShowAnswer

   B — The derivative's shape is the output's shape followed by the input's shape — so a scalar of a vector is 1 by n, a vector of a vector is m by n, and a matrix of a vector is m by n by n — Every dimension error in the chapter is a violation of that one line. Stating the shape before computing anything turns the rest into bookkeeping, which is exactly why Exercises 5.7 and 5.8 ask for the dimensions explicitly.

2. Why does the book make the gradient a ROW vector in Equation 5.40?

   A Convention inherited from physics B So the chain rule is a plain left-to-right matrix product with the shapes lining up, and it generalises to vector-valued functions without changing conventions — a 1 by n times an n by 1 is the scalar you want C To save memory D Because gradients are covectors
   pch.quizShowAnswer

   B — So the chain rule is a plain left-to-right matrix product with the shapes lining up, and it generalises to vector-valued functions without changing conventions — a 1 by n times an n by 1 is the scalar you want — The book gives both reasons at Equation 5.40. The practical consequence appears at Eq 5.148, where grad f times (x - x0) needs no transpose at all.

3. Equation 5.158 gives the second-order Taylor term as trace(H delta delta-transpose) and as delta-transpose H delta. Which should you compute?

   A The trace form, it is more general B The quadratic form. Both give the same number, but the trace form builds a D by D rank-one intermediate — at D = 1000000 that is 4 TB allocated to produce one scalar C Either, they cost the same D Neither, use einsum
   pch.quizShowAnswer

   B — The quadratic form. Both give the same number, but the trace form builds a D by D rank-one intermediate — at D = 1000000 that is 4 TB allocated to produce one scalar — The trace form is the right way to STATE the term, because it generalises to arbitrary k. The quadratic form is the right way to compute it: two matrix-vector products and nothing bigger than D.

4. What do the four outcomes of the second-order test tell you when an eigenvalue is zero?

   A That the point is not an optimum B Nothing at all — ask a higher derivative. x1^2+x2^4, x1^2+x2^3 and x1^2-x2^4 share the Hessian [[2,0],[0,0]] at the origin and are respectively a strict minimum and two non-optima C That the point is a saddle D That the Hessian was computed wrongly
   pch.quizShowAnswer

   B — Nothing at all — ask a higher derivative. x1^2+x2^4, x1^2+x2^3 and x1^2-x2^4 share the Hessian [[2,0],[0,0]] at the origin and are respectively a strict minimum and two non-optima — And numerically it is worse: the differencing floor is +2h^2, so a singular Hessian tests as positive definite at every step size. Compare eigenvalues against a tolerance scaled to the entries, never against zero.

5. Reverse-mode AD costs about 2.67 times the forward pass on Example 5.14. Why is a constant factor the interesting part?

   A Because 2.67 is small B Because the naive expectation is a blow-up: Equation 5.110's explicit derivative is 11 operations and recomputes x-squared and exp(x-squared). A constant factor is what makes training feasible at all, and it comes from reusing stored forward values C Because it beats finite differences D Because forward mode is 3 times slower
   pch.quizShowAnswer

   B — Because the naive expectation is a blow-up: Equation 5.110's explicit derivative is 11 operations and recomputes x-squared and exp(x-squared). A constant factor is what makes training feasible at all, and it comes from reusing stored forward values — The book calls this counterintuitive itself. The mechanism is reuse: the reverse sweep reads the forward values instead of recomputing them, which is exactly the 'unnecessary overhead' symbolic differentiation pays.

Recall card

• The shape rule subsumes the chapter: the derivative’s shape is the output’s shape followed by the input’s. Scalar of a vector is 1 by n, vector of a vector is m by n, matrix of a vector is m by n by n.
• Equation 5.40 makes the gradient a ROW, so the chain rule is a left-to-right matrix product and Equation 5.148 needs no transpose.
• Equation 5.32’s chain rule is the only rule in the chapter. §5.2, §5.3, §5.6 and §5.8 are all it, at increasing levels of generality.
• Equation 5.56’s Jacobian is m by n, and det J is the local volume scale factor — local only, and meaningless where it changes sign.
• Equation 5.106: the derivative of a quadratic form is x-transpose (B + B-transpose), not 2 x-transpose B. Measured cost of the shortcut on a non-symmetric B: 62.5 percent.
• Equation 5.145 sums over a node’s children. Example 5.14’s node a is the only one with two, and dropping the second term costs 9.5 to 47.8 percent, worst for small x.
• Reverse mode is one sweep per OUTPUT, forward mode one per INPUT. Crossover at n = 3 for a scalar loss, and the reverse sweep costs 2.67 times the forward pass on Example 5.14 — a constant, not a blow-up.
• AD has no step size: 2.2e-16 against the best of 200 central differences at 9.8e-12.
• Equation 5.146 needs its hypothesis — twice CONTINUOUSLY differentiable. The standard counterexample’s two iterated limits at the origin are -1 and +1, gap exactly 2.
• Equation 5.147’s Hessian is symmetric, so §4.2 applies, and the curvature along a unit d is d-transpose H d, squeezed between the eigenvalues and attaining both.
• The second-order test has FOUR outcomes, and the fourth — some eigenvalue zero — means ask a higher derivative. Three functions share [[2,0],[0,0]] and have three different answers.
• Never test an eigenvalue against zero: the second-difference floor is exactly 2h^2 and it is positive.
• Newton’s method solves grad f = 0, not “find a minimum”. 8 steps against gradient descent’s 426 on a convex test, and exactly onto the saddle on x1^2 - x2^2.
• The condition number is the price gradient descent pays: 18, 94, 939, 9384 steps for kappa of 2, 10, 100, 1000.
• Definition 5.7’s delta-to-the-k is a k-fold OUTER product, and the order-k tensor has D^k entries — 1e12 at a million parameters, which is where practice stops.
• Taylor error goes like r to the (n+1) as r goes to zero, measured 1.067, 2.005, 3.019 — and on a fixed window a higher order can be five times worse.

Next: Vector Calculus Overview for the map of the chapter, or on to Chapter 6, Probability and Distributions.