Useful Identities for Computing Gradients

§5.5 is a page of the book and a table of ten equations. It states them, cites Petersen and Pedersen’s Matrix Cookbook, and proves none of them.

That is a reasonable choice for a textbook and a bad situation for a reader, because a memorised identity you cannot check is worse than no identity at all — you will use it, get a plausible number, and not find out. So this page does two things: verifies all ten against independent finite differences, and isolates the two places where the stated form differs from the shortcut people actually write, with the size of the error when they do.

Both of those are about symmetry, and both are off by tens of percent.

What you’ll learn

• All ten identities, Equations 5.99 to 5.108, each verified numerically.
• Why Equation 5.107 says $\mathbf{x}^\top(\mathbf{B}+\mathbf{B}^\top)$ and not $2\mathbf{x}^\top\mathbf{B}$ — with the 29% error the shortcut costs.
• Why Equation 5.108 carries the clause “for symmetric $\mathbf{W}$” — with the 107% error when it is not.
• Which identities are the same statement in different clothes, and which are genuinely independent.
• The cost question the section raises without answering: if a library differentiates anything, why memorise these? Measured.
• The book’s own caveat about traces and transposes of higher-order tensors.

Intuition: the identities are differentials

Every line in §5.5 is a differential — a statement about what happens to the output when you nudge the input — rather than a four-index tensor. That is why they are usable, and it is the whole reason the section exists directly after §5.4’s tensors.

Read $\partial f(\mathbf{X})^{-1}/\partial\mathbf{X} = -f(\mathbf{X})^{-1}\,\partial f(\mathbf{X})\,f(\mathbf{X})^{-1}$ as: nudge $\mathbf{X}$, and the inverse moves by minus the inverse, times the nudge in $f$, times the inverse again. No index ever appears. Contracting §5.4’s tensor against a direction gives exactly this, which the previous page measured.

diagram Diagram mermaid

  flowchart TD
  T["§5.4: the derivative is a
fourth-order tensor"]
  T -->|"contract with a direction"| D["a DIFFERENTIAL:
a statement about one nudge"]
  D --> L["§5.5's ten identities
Eq 5.99 – 5.108"]
  L --> STRUCT["structural, about any f:
5.99 transpose · 5.100 trace
5.101 determinant · 5.102 inverse"]
  L --> CONC["concrete, for specific forms:
5.103 aᵀX⁻¹b · 5.104/5.105 aᵀx
5.106 aᵀXb · 5.107 xᵀBx · 5.108 least squares"]
  CONC --> TRAP1["5.107: (B + Bᵀ), not 2B
the shortcut is 29% wrong"]
  CONC --> TRAP2["5.108: 'for symmetric W'
without it, 107% wrong"]
  TRAP1 --> WHY["both traps are the same fact:
a quadratic form only sees
the symmetric part of its matrix"]
  TRAP2 --> WHY
  L --> CAVEAT["the book's Remark: trace and
transpose are not defined for
higher-order tensors — that is
a tensor contraction"]

The math

The ten identities

Equations 5.99 to 5.108

Here $\mathrm{tr}(\cdot)$ is the trace (Definition 4.4), $\det(\cdot)$ the determinant (§4.1), and $f(\mathbf{X})^{-1}$ the inverse of $f(\mathbf{X})$, assuming it exists.

$$\frac{\partial}{\partial\mathbf{X}}f(\mathbf{X})^\top = \left(\frac{\partial f(\mathbf{X})}{\partial\mathbf{X}}\right)^\top \tag{5.99}$$$$\frac{\partial}{\partial\mathbf{X}}\mathrm{tr}\bigl(f(\mathbf{X})\bigr) = \mathrm{tr}\left(\frac{\partial f(\mathbf{X})}{\partial\mathbf{X}}\right) \tag{5.100}$$$$\frac{\partial}{\partial\mathbf{X}}\det\bigl(f(\mathbf{X})\bigr) = \det\bigl(f(\mathbf{X})\bigr)\,\mathrm{tr}\left(f(\mathbf{X})^{-1}\frac{\partial f(\mathbf{X})}{\partial\mathbf{X}}\right) \tag{5.101}$$$$\frac{\partial}{\partial\mathbf{X}}f(\mathbf{X})^{-1} = -f(\mathbf{X})^{-1}\frac{\partial f(\mathbf{X})}{\partial\mathbf{X}}f(\mathbf{X})^{-1} \tag{5.102}$$$$\frac{\partial\,\mathbf{a}^\top\mathbf{X}^{-1}\mathbf{b}}{\partial\mathbf{X}} = -(\mathbf{X}^{-1})^\top\mathbf{a}\mathbf{b}^\top(\mathbf{X}^{-1})^\top \tag{5.103}$$$$\frac{\partial\,\mathbf{x}^\top\mathbf{a}}{\partial\mathbf{x}} = \mathbf{a}^\top \tag{5.104}$$$$\frac{\partial\,\mathbf{a}^\top\mathbf{x}}{\partial\mathbf{x}} = \mathbf{a}^\top \tag{5.105}$$$$\frac{\partial\,\mathbf{a}^\top\mathbf{X}\mathbf{b}}{\partial\mathbf{X}} = \mathbf{a}\mathbf{b}^\top \tag{5.106}$$$$\frac{\partial\,\mathbf{x}^\top\mathbf{B}\mathbf{x}}{\partial\mathbf{x}} = \mathbf{x}^\top(\mathbf{B} + \mathbf{B}^\top) \tag{5.107}$$$$\frac{\partial}{\partial\mathbf{s}}(\mathbf{x} - \mathbf{A}\mathbf{s})^\top\mathbf{W}(\mathbf{x} - \mathbf{A}\mathbf{s}) = -2(\mathbf{x}-\mathbf{A}\mathbf{s})^\top\mathbf{W}\mathbf{A} \quad\text{for symmetric }\mathbf{W} \tag{5.108}$$

The book’s Remark on tensors

In this book, we only cover traces and transposes of matrices. However, we have seen that derivatives can be higher-dimensional tensors, in which case the usual trace and transpose are not defined. In these cases, the trace of a $D\times D\times E\times F$ tensor would be an $E\times F$-dimensional matrix. This is a special case of a tensor contraction.

That remark is doing real work. Equations 5.99 to 5.102 are written with $\mathrm{tr}$ and $\top$ applied to $\partial f/\partial\mathbf{X}$, which for a matrix argument is a tensor — so those symbols mean contractions, not the matrix operations they look like. In practice you apply the identity in a direction, where everything is a matrix again.

Which of these are actually different statements

Ten equations, fewer ideas.

identities	what they really are
5.104, 5.105	the same identity. $\mathbf{x}^\top\mathbf{a} = \mathbf{a}^\top\mathbf{x}$ because an inner product is symmetric, so there is one fact here, listed twice for lookup convenience
5.106	5.104 with a matrix in the middle. Setting $\mathbf{a},\mathbf{b}$ to basis vectors picks out a single entry
5.103	5.106 composed with 5.102: differentiate $\mathbf{X}^{-1}$, then sandwich
5.107	the only genuinely quadratic one, and the only one where symmetry appears explicitly
5.108	5.107 composed with the chain rule and a linear inner function — and it is Chapter 9’s gradient
5.99–5.102	structural: true for any differentiable $f$, not for a specific form

So the list is four structural rules and three independent concrete ones, with the rest following by composition. Worth knowing, because it tells you which ones to memorise: 5.102 and 5.107, and 5.108 because you will use it constantly.

The two traps

Both are the same fact wearing different hats: a quadratic form only ever sees the symmetric part of its matrix.

For any $\mathbf{B}$, write $\mathbf{B} = \mathbf{B}_s + \mathbf{B}_a$ with $\mathbf{B}_s = \tfrac12(\mathbf{B}+\mathbf{B}^\top)$ symmetric and $\mathbf{B}_a = \tfrac12(\mathbf{B}-\mathbf{B}^\top)$ antisymmetric. Then

$$\mathbf{x}^\top\mathbf{B}_a\mathbf{x} = 0 \quad\text{for every }\mathbf{x}$$

because $\mathbf{x}^\top\mathbf{B}_a\mathbf{x}$ is its own negative transpose. So $\mathbf{x}^\top\mathbf{B}\mathbf{x} = \mathbf{x}^\top\mathbf{B}_s\mathbf{x}$, and differentiating gives $2\mathbf{x}^\top\mathbf{B}_s = \mathbf{x}^\top(\mathbf{B}+\mathbf{B}^\top)$ — Equation 5.107.

Trap one. Writing $2\mathbf{x}^\top\mathbf{B}$ instead is right precisely when $\mathbf{B}$ is symmetric. Measured on a random $4\times4$:

form	relative error against a finite difference
$\mathbf{x}^\top(\mathbf{B}+\mathbf{B}^\top)$	$8.8\times10^{-11}$
$2\mathbf{x}^\top\mathbf{B}$	$\mathbf{1.20}$

120% wrong — the error is larger than the gradient itself. And with a symmetric $\mathbf{B}$ the two forms agree to 0.0e+00, so the bug is invisible on every symmetric test case you might try first.

Trap two. Equation 5.108’s clause “for symmetric $\mathbf{W}$” is not decoration. The general result is

$$\frac{\partial}{\partial\mathbf{s}}(\mathbf{x}-\mathbf{A}\mathbf{s})^\top\mathbf{W}(\mathbf{x}-\mathbf{A}\mathbf{s}) = -(\mathbf{x}-\mathbf{A}\mathbf{s})^\top(\mathbf{W}+\mathbf{W}^\top)\mathbf{A}$$

and the factor of $2$ appears only when $\mathbf{W} = \mathbf{W}^\top$. Measured:

$\mathbf{W}$	Equation 5.108 as stated	the general form
symmetric	$1.3\times10^{-10}$	$1.3\times10^{-10}$
$\mathbf{W} = \mathbf{I}$	$8.7\times10^{-11}$	$8.7\times10^{-11}$
not symmetric	$\mathbf{2.9\times10^{-1}}$	$6.6\times10^{-12}$

Read the first two rows across: for a symmetric $\mathbf{W}$ the stated form and the general form give the same number, because $\mathbf{W}+\mathbf{W}^\top = 2\mathbf{W}$ exactly. No symmetric test can tell them apart. On the third row the stated form is $29\%$ wrong while the general form stays at $6.6\times10^{-12}$.

(The figure below uses a different matrix and gets $1.07$ — $107\%$ — for the same failure. How wrong the shortcut is depends on how far from symmetric $\mathbf{W}$ is; that it is wrong does not.)

In practice $\mathbf{W}$ is usually symmetric — it is an inverse covariance, or the identity — which is exactly why the clause gets forgotten.

From the book

Section 5.5 lists ten gradients “frequently required in a machine learning context”, citing Petersen and Pedersen (2012), and uses $\mathrm{tr}(\cdot)$ from Definition 4.4, $\det(\cdot)$ from §4.1 and $f(\mathbf{X})^{-1}$ assuming it exists. Equations 5.99 to 5.102 are structural rules for the transpose, trace, determinant and inverse of a general $f(\mathbf{X})$. Equation 5.103 gives $\partial(\mathbf{a}^\top\mathbf{X}^{-1}\mathbf{b})/\partial\mathbf{X}$; 5.104 and 5.105 give $\partial(\mathbf{x}^\top\mathbf{a})/\partial\mathbf{x} = \mathbf{a}^\top = \partial(\mathbf{a}^\top\mathbf{x})/\partial\mathbf{x}$; 5.106 gives $\mathbf{a}\mathbf{b}^\top$; 5.107 gives $\mathbf{x}^\top(\mathbf{B}+\mathbf{B}^\top)$; and 5.108 gives the least-squares gradient $-2(\mathbf{x}-\mathbf{A}\mathbf{s})^\top\mathbf{W}\mathbf{A}$ for symmetric $\mathbf{W}$. A Remark then notes that the book covers only traces and transposes of matrices, that these are undefined for higher-order tensors, and that the trace of a $D\times D\times E\times F$ tensor would be an $E\times F$ matrix — a special case of a tensor contraction.

Worked example by hand

Equation 5.107, derived twice

By coordinates. $\mathbf{x}^\top\mathbf{B}\mathbf{x} = \sum_{i}\sum_{j}x_iB_{ij}x_j$. Differentiate with respect to $x_k$: the variable $x_k$ appears once as the $i$ index and once as the $j$ index, so the product rule gives two terms,

$$\frac{\partial}{\partial x_k}\sum_{i,j}x_iB_{ij}x_j = \sum_{j}B_{kj}x_j + \sum_{i}x_iB_{ik} = (\mathbf{B}\mathbf{x})_k + (\mathbf{B}^\top\mathbf{x})_k$$

and collecting into a row gives $\mathbf{x}^\top(\mathbf{B}+\mathbf{B}^\top)$. The two terms are where the transpose comes from — one for each occurrence of $x_k$.

By differential. Nudge $\mathbf{x}$ by $\mathrm{d}\mathbf{x}$:

$$(\mathbf{x}+\mathrm{d}\mathbf{x})^\top\mathbf{B}(\mathbf{x}+\mathrm{d}\mathbf{x}) = \mathbf{x}^\top\mathbf{B}\mathbf{x} + \mathrm{d}\mathbf{x}^\top\mathbf{B}\mathbf{x} + \mathbf{x}^\top\mathbf{B}\,\mathrm{d}\mathbf{x} + O(\mathrm{d}\mathbf{x}^2)$$

The two first-order terms are scalars, so each equals its own transpose: $\mathrm{d}\mathbf{x}^\top\mathbf{B}\mathbf{x} = \mathbf{x}^\top\mathbf{B}^\top\mathrm{d}\mathbf{x}$. Adding them,

$$\mathrm{d}\bigl(\mathbf{x}^\top\mathbf{B}\mathbf{x}\bigr) = \mathbf{x}^\top(\mathbf{B}+\mathbf{B}^\top)\,\mathrm{d}\mathbf{x}$$

Two lines, no indices. This is the method to use.

Equation 5.108, from 5.107 and the chain rule

Let $\mathbf{e}(\mathbf{s}) = \mathbf{x} - \mathbf{A}\mathbf{s}$ and $L(\mathbf{e}) = \mathbf{e}^\top\mathbf{W}\mathbf{e}$. Then by §5.3’s chain rule,

$$\frac{\partial L}{\partial\mathbf{s}} = \underbrace{\frac{\partial L}{\partial\mathbf{e}}}_{1\times N}\underbrace{\frac{\partial\mathbf{e}}{\partial\mathbf{s}}}_{N\times D} = \mathbf{e}^\top(\mathbf{W}+\mathbf{W}^\top)\cdot(-\mathbf{A})$$

using Equation 5.107 for the first factor and Example 5.9 for the second. If $\mathbf{W} = \mathbf{W}^\top$ this collapses to $-2\mathbf{e}^\top\mathbf{W}\mathbf{A}$, which is Equation 5.108. And with $\mathbf{W} = \mathbf{I}$ it is Example 5.11’s $-2\mathbf{e}^\top\boldsymbol{\Phi}$ — the same identity, three times, at increasing generality.

Equation 5.102, and why the minus sign

From $f\,f^{-1} = \mathbf{I}$, differentiate both sides. The right side is constant, so

$$\mathrm{d}f\cdot f^{-1} + f\cdot\mathrm{d}\bigl(f^{-1}\bigr) = \mathbf{0} \implies \mathrm{d}\bigl(f^{-1}\bigr) = -f^{-1}\,\mathrm{d}f\,f^{-1}$$

which is Equation 5.102. The minus sign is not a convention — it comes from moving a term across an equals sign, and the two $f^{-1}$ factors come from multiplying through on both sides. In one dimension this is $\mathrm{d}(1/x) = -\mathrm{d}x/x^2$, which is the same statement with the order irrelevant.

All ten, checked

ten_identities.py

import numpy as np
 
def numeric_jac(f, x, h=1e-6):
    """Central-difference Jacobian of a flattened function of a flat input."""
    x = np.asarray(x, dtype=float).ravel()
    f0 = np.asarray(f(x)).ravel()
    J = np.zeros((f0.size, x.size))
    for j in range(x.size):
        e = np.zeros_like(x)
        e[j] = h
        J[:, j] = (np.asarray(f(x + e)).ravel() - np.asarray(f(x - e)).ravel()) / (2 * h)
    return J
 
rng = np.random.default_rng(19)
n, h = 3, 1e-6
fX = lambda M: M @ M                        # any smooth matrix function
tests = []
 
# 5.99  d f(X)^T = (d f(X))^T, in a direction H
X, H = rng.normal(size=(n, n)), rng.normal(size=(n, n))
tests.append(("5.99", (fX(X + h*H).T - fX(X - h*H).T) / (2*h),
                      ((fX(X + h*H) - fX(X - h*H)) / (2*h)).T))
 
# 5.100  d tr(f) = tr(d f)
tests.append(("5.100", np.array([[(np.trace(fX(X + h*H)) - np.trace(fX(X - h*H))) / (2*h)]]),
                       np.array([[np.trace((fX(X + h*H) - fX(X - h*H)) / (2*h))]])))
 
# 5.101  d det(f) = det(f) tr(f^-1 d f)
Xp, Hp = rng.normal(size=(n, n)) + 3*np.eye(n), rng.normal(size=(n, n))
dfp = (fX(Xp + h*Hp) - fX(Xp - h*Hp)) / (2*h)
tests.append(("5.101",
    np.array([[(np.linalg.det(fX(Xp + h*Hp)) - np.linalg.det(fX(Xp - h*Hp))) / (2*h)]]),
    np.array([[np.linalg.det(fX(Xp)) * np.trace(np.linalg.inv(fX(Xp)) @ dfp)]])))
 
# 5.102  d f^-1 = -f^-1 (d f) f^-1
tests.append(("5.102", (np.linalg.inv(fX(Xp + h*Hp)) - np.linalg.inv(fX(Xp - h*Hp))) / (2*h),
                       -np.linalg.inv(fX(Xp)) @ dfp @ np.linalg.inv(fX(Xp))))
 
# 5.103  d(a^T X^-1 b)/dX = -(X^-1)^T a b^T (X^-1)^T
Xi = rng.normal(size=(n, n)) + 3*np.eye(n)
a, b = rng.normal(size=n), rng.normal(size=n)
Xinv = np.linalg.inv(Xi)
tests.append(("5.103", numeric_jac(lambda v: np.array([a @ np.linalg.inv(v.reshape(n, n)) @ b]), Xi),
                       (-Xinv.T @ np.outer(a, b) @ Xinv.T).reshape(1, -1)))
 
# 5.104 and 5.105  d(x^T a)/dx = a^T = d(a^T x)/dx
x, av = rng.normal(size=n), rng.normal(size=n)
tests.append(("5.104", numeric_jac(lambda v: np.array([v @ av]), x), av.reshape(1, -1)))
tests.append(("5.105", numeric_jac(lambda v: np.array([av @ v]), x), av.reshape(1, -1)))
 
# 5.106  d(a^T X b)/dX = a b^T
Xm = rng.normal(size=(n, n))
tests.append(("5.106", numeric_jac(lambda v: np.array([a @ v.reshape(n, n) @ b]), Xm),
                       np.outer(a, b).reshape(1, -1)))
 
# 5.107  d(x^T B x)/dx = x^T (B + B^T)
B = rng.normal(size=(n, n))
tests.append(("5.107", numeric_jac(lambda v: np.array([v @ B @ v]), x),
                       (x @ (B + B.T)).reshape(1, -1)))
 
# 5.108  d/ds (x - As)^T W (x - As) = -2 (x - As)^T W A,  W symmetric
m = 4
Aw = rng.normal(size=(m, n))
W0 = rng.normal(size=(m, m)); W = W0 + W0.T
xw, s0 = rng.normal(size=m), rng.normal(size=n)
r = xw - Aw @ s0
tests.append(("5.108", numeric_jac(lambda v: np.array([(xw - Aw @ v) @ W @ (xw - Aw @ v)]), s0),
                       (-2 * r @ W @ Aw).reshape(1, -1)))
 
errs = []
for name, num, ana in tests:
    num = np.asarray(num, float)
    ana = np.asarray(ana, float).reshape(num.shape)
    e = float(np.abs(num - ana).max()) / max(float(np.abs(ana).max()), 1e-12)
    errs.append(e)
    print(f"  {name:>7}  relative error {e:.2e}")
print(f"  worst of the ten: {max(errs):.1e}")

output

     5.99  relative error 0.00e+00
    5.100  relative error 2.51e-11
    5.101  relative error 1.29e-10
    5.102  relative error 4.71e-11
    5.103  relative error 2.71e-10
    5.104  relative error 4.00e-11
    5.105  relative error 4.00e-11
    5.106  relative error 5.25e-11
    5.107  relative error 8.70e-11
    5.108  relative error 1.46e-10
  worst of the ten: 2.7e-10

Two readings.

Equation 5.99’s error is exactly 0.00e+00. That is not machine precision, it is identically zero — the two sides are the same four numbers rearranged, because transposing before or after a difference of transposes is the same set of subtractions. A check that comes out exactly zero is checking the algebra rather than the analysis.

The other nine are between $2.5\times10^{-11}$ and $2.7\times10^{-10}$, which is the test’s floor rather than the identities’. §5.1’s figure showed a central difference at $h = 10^{-6}$ cannot do better than about $10^{-10}$ relative. So these say “consistent with exact”, and the two traps below sit at $10^{-1}$ and $10^{0}$ — nine orders away.

See it move

sketch A quadratic form only sees the symmetric part p5.js Pause Restart

Drag the entries of B and watch two things: the antisymmetric part's contribution to x-transpose B x, which is always exactly zero, and the gap between the correct gradient and the 2 x-transpose B shortcut. Press 'symmetrise' and the gap vanishes — which is why the shortcut survives every symmetric test case.

sketch Identity, autodiff, or differencing? p5.js Pause Restart

The cost of one gradient of x-transpose B x, three ways, as the dimension grows. Drag n. The closed form and reverse-mode autodiff track each other; finite differencing needs 2n function evaluations and pulls away as exactly n+1 times the cost. The identity is not faster than autodiff — it is the same order, and that is the point.

From scratch

the_two_traps.py

import numpy as np
 
def numeric_jac(f, x, h=1e-6):
    x = np.asarray(x, dtype=float).ravel()
    f0 = np.asarray(f(x)).ravel()
    J = np.zeros((f0.size, x.size))
    for j in range(x.size):
        e = np.zeros_like(x)
        e[j] = h
        J[:, j] = (np.asarray(f(x + e)).ravel() - np.asarray(f(x - e)).ravel()) / (2 * h)
    return J
 
rng = np.random.default_rng(31)
 
# ---- Trap one: Eq 5.107 is (B + B^T), not 2B.
print("Trap one: d(x^T B x)/dx")
B = rng.normal(size=(4, 4))
x = rng.normal(size=4)
num = numeric_jac(lambda v: np.array([v @ B @ v]), x)
right = (x @ (B + B.T)).reshape(1, -1)
wrong = (2 * x @ B).reshape(1, -1)
scale = float(np.abs(right).max())
print(f"  x^T(B + B^T)   relative error {float(np.abs(num - right).max()) / scale:.2e}")
print(f"  2 x^T B        relative error {float(np.abs(num - wrong).max()) / scale:.2e}")
 
Bs = B + B.T                                   # now symmetric
num2 = numeric_jac(lambda v: np.array([v @ Bs @ v]), x)
r2 = (x @ (Bs + Bs.T)).reshape(1, -1)
w2 = (2 * x @ Bs).reshape(1, -1)
print(f"  with a SYMMETRIC B the two forms differ by {float(np.abs(r2 - w2).max()):.1e}")
print(f"  -- so the shortcut passes every symmetric test case")
 
# And why: the antisymmetric part contributes nothing to the form itself.
Ba = (B - B.T) / 2
vals = [float(v @ Ba @ v) for v in rng.normal(size=(5, 4))]
print(f"  x^T B_a x for five random x: {[f'{v:.1e}' for v in vals]}")
 
# ---- Trap two: Eq 5.108 needs W symmetric.
print()
print("Trap two: d/ds (x - As)^T W (x - As)")
m, d = 6, 3
A = rng.normal(size=(m, d))
xv = rng.normal(size=m)
s0 = rng.normal(size=d)
e0 = xv - A @ s0
 
for label, W in (("symmetric", (lambda M: M + M.T)(rng.normal(size=(m, m)))),
                 ("identity", np.eye(m)),
                 ("NOT symmetric", rng.normal(size=(m, m)))):
    num = numeric_jac(lambda v: np.array([(xv - A @ v) @ W @ (xv - A @ v)]), s0)
    stated = (-2 * e0 @ W @ A).reshape(1, -1)              # Eq 5.108 as written
    general = (-e0 @ (W + W.T) @ A).reshape(1, -1)         # the general form
    sc = max(float(np.abs(num).max()), 1e-12)
    print(f"  W {label:14} Eq 5.108 as stated {float(np.abs(num - stated).max())/sc:9.2e}"
          f"   general form {float(np.abs(num - general).max())/sc:9.2e}")

output

Trap one: d(x^T B x)/dx
  x^T(B + B^T)   relative error 8.78e-11
  2 x^T B        relative error 1.20e+00
  with a SYMMETRIC B the two forms differ by 0.0e+00
  -- so the shortcut passes every symmetric test case
  x^T B_a x for five random x: ['3.5e-18', '-4.2e-16', '3.5e-18', '-5.4e-17', '-1.4e-16']
 
Trap two: d/ds (x - As)^T W (x - As)
  W symmetric      Eq 5.108 as stated  1.26e-10   general form  1.26e-10
  W identity       Eq 5.108 as stated  8.70e-11   general form  8.70e-11
  W NOT symmetric  Eq 5.108 as stated  2.86e-01   general form  6.57e-12

The two rows to read are the last two.

For a symmetric $\mathbf{W}$ the stated identity and the general form give the same answer — because $\mathbf{W}+\mathbf{W}^\top = 2\mathbf{W}$ exactly. So there is no test with a symmetric $\mathbf{W}$ that can distinguish them, and inverse covariances are symmetric, and the identity matrix is symmetric.

For a non-symmetric $\mathbf{W}$ the stated identity is off by $2.9\times10^{-1}$ while the general form stays at $6.6\times10^{-12}$ — four orders apart on the same input. The clause “for symmetric $\mathbf{W}$” is the whole difference.

And the last line of the first block explains both traps: $\mathbf{x}^\top\mathbf{B}_a\mathbf{x}$ comes out between $3.5\times10^{-18}$ and $4.2\times10^{-16}$ for five random $\mathbf{x}$ — zero to within rounding. The antisymmetric part is invisible to the form, so it cannot appear in the derivative, so any expression that retains it is wrong by exactly the amount it retains.

On real data

figure All ten identities of §5.5, checked rather than quoted matplotlib

Every rule differenced independently at h = 1e-6. Equation 5.99 comes out at exactly 0.0e+00 because both sides are the same numbers rearranged; the other nine sit between 2.5e-11 and 2.7e-10, which is the central difference's own floor rather than the identities' error.

figure Why memorise an identity when a library will differentiate anything matplotlib

The closed form and reverse-mode AD are the same order and stay parallel. Finite differencing needs 2n evaluations of an n-squared form, so its ratio to the closed form is exactly n + 1 — 3x at n = 2, 257x at n = 256. The identity's advantage over AD is not speed, it is needing no graph.

figure Equation 5.108's symmetry clause is load-bearing matplotlib

Left, the residuals the weighted least-squares objective squares. Right, the identity as stated: 1.7e-10 for a symmetric W, 5.6e-11 for the identity matrix, and 1.07 relative — larger than the gradient — for a non-symmetric one. The general form is minus e-transpose times (W plus W-transpose) times A.

Reading the plot

From the ten-identities figure. The bars span $2.5\times10^{-11}$ to $2.7\times10^{-10}$, and the useful thing about that narrow range is that it is uninformative about the identities: it is the resolution of the test. What would be informative is a bar at $10^{-1}$, and the next figure has one.

Two bars are worth naming. Equation 5.99 at exactly 0.0e+00 — the algebra, not the analysis, as discussed above. And Equation 5.103 at $2.7\times10^{-10}$, the worst of the ten, which is expected: it differentiates through a matrix inverse, and inverting a $3\times3$ amplifies input perturbations by its condition number, so the difference quotient has further to fall before it converges.

From the cost figure. The green and blue lines being parallel is the answer to the question the section raises. A closed-form identity is $2n^2$ multiplications and reverse-mode autodiff is about $3n^2$ — the same order, a constant factor apart. So memorising identities does not buy asymptotic speed over a library.

What it buys is visible in the red line and in the ratio:

$n$	closed form	reverse AD	differencing	ratio
$2$	$8$	$12$	$24$	$3\times$
$8$	$128$	$192$	$1152$	$9\times$
$32$	$2048$	$3072$	$67\,584$	$33\times$
$128$	$32\,768$	$49\,152$	$4\,227\,072$	$129\times$
$256$	$131\,072$	$196\,608$	$33\,685\,504$	$257\times$

The ratio is exactly $n+1$, which falls straight out of the arithmetic: $\frac{2n(n^2+n)}{2n^2} = \frac{n^2+n}{n} = n+1$. So differencing is not a constant factor worse, it is a factor of the dimension worse — which is the real reason nobody gradient-checks a large model except on a tiny slice.

And the honest summary of the identities: they are for deriving on paper. When you are writing code, use the library; when you are working out what the gradient is, these ten lines are what you have.

From the symmetry figure. The right panel is the only bar chart in this module with a bar at $10^{0}$: $1.7\times10^{-10}$ for a symmetric $\mathbf{W}$, $5.6\times10^{-11}$ for the identity, and $1.07$ — $107\%$ — for a non-symmetric one. The failing case is the one you would never test, because $\mathbf{W}$ is almost always symmetric.

The left panel is there to say what the objective is: those vertical red segments are the residuals, and $\mathbf{W}$ decides how much each one counts. With $\mathbf{W} = \mathbf{I}$ they count equally, which is ordinary least squares; with $\mathbf{W} = \boldsymbol{\Sigma}^{-1}$ they are weighted by certainty, which is Chapter 9’s generalised least squares. Both symmetric. The clause survives because the case that violates it does not arise — until someone writes W = np.random.randn(m, m) in a test.

Pitfalls

Equation 5.107 is (B + B-transpose), not 2B

Measured on a random $4\times4$: the stated form has relative error $8.8\times10^{-11}$ and $2\mathbf{x}^\top\mathbf{B}$ has $\mathbf{1.20}$ — 120% wrong, a larger error than the gradient is large. And for symmetric $\mathbf{B}$ the two agree to 0.0e+00, so the shortcut passes every symmetric test case. If a Hessian or gradient is mysteriously wrong only on asymmetric input, this is why.

Equation 5.108’s ‘for symmetric W’ is not decoration

With a non-symmetric $\mathbf{W}$ the stated identity is off by $2.9\times10^{-1}$ relative in one test and $1.07$ in the figure’s — how wrong depends on how far from symmetric $\mathbf{W}$ is. The general form is $-\mathbf{e}^\top(\mathbf{W}+\mathbf{W}^\top)\mathbf{A}$, which stays at $6.6\times10^{-12}$ on the same input. Since $\mathbf{W}$ is usually an inverse covariance or the identity, the clause is almost always satisfied and therefore almost never checked.

The trace and transpose in Equations 5.99 to 5.102 mean contractions

The book’s own Remark: for higher-order tensors the usual trace and transpose are undefined, and the trace of a $D\times D\times E\times F$ tensor would be an $E\times F$ matrix — a tensor contraction. So when $f$ takes a matrix argument, read those symbols as contractions over the appropriate index pair, or apply the identity in a direction where everything is a matrix again.

Equations 5.104 and 5.105 are the same identity

$\mathbf{x}^\top\mathbf{a} = \mathbf{a}^\top\mathbf{x}$ because an inner product is symmetric, so those two lines are one fact listed twice for lookup. Counting them as two independent results overstates how much §5.5 contains: it is four structural rules and three concrete ones, with the rest following by composition.

Do not use these identities as an implementation strategy

The closed form and reverse-mode autodiff are the same order — $2n^2$ against $3n^2$ — so hand-derived gradients buy you no asymptotic speed and cost you a maintenance burden every time the model changes. Use them to understand the gradient and to sanity-check a library’s answer on a small case. Let the library compute it.

Compare

identity	the shortest way to see it	where it is used
5.99 transpose	both sides are the same entries reindexed	rearranging derivations
5.100 trace	the trace is a sum, and differentiation is linear	Ch 6 Gaussian log-densities
5.101 determinant	Jacobi’s formula; from $\log\det$ and 5.102	Ch 6 change of variables, normalising flows
5.102 inverse	differentiate $ff^{-1} = \mathbf{I}$ and move a term	Ch 9 posterior updates
5.103	5.106 composed with 5.102	Gaussian conditionals
5.104 = 5.105	an inner product is symmetric	everywhere
5.106	5.104 with a matrix inserted	picking out matrix entries
5.107 quadratic	two terms because $x_k$ appears twice	every quadratic loss, every Hessian
5.108 least squares	5.107 then the chain rule then Example 5.9	Ch 9’s starting point

The three in bold are the ones to memorise. The rest you can rebuild from them in a line or two, which is a better use of memory than a lookup table you cannot check.

pch.quizTag Check your understanding

1. Why does Equation 5.107 say x-transpose times (B plus B-transpose) rather than 2 x-transpose B?

   A Convention B Because differentiating the double sum hits x_k twice — once as the row index and once as the column index — giving Bx plus B-transpose x. Writing 2B keeps the antisymmetric part, which the quadratic form cannot see C Because B might not be invertible D Because the gradient is a row vector
   pch.quizShowAnswer

   B — Because differentiating the double sum hits x_k twice — once as the row index and once as the column index — giving Bx plus B-transpose x. Writing 2B keeps the antisymmetric part, which the quadratic form cannot see — Measured on a random 4x4: the shortcut is off by 1.20 relative — 120 percent, a larger error than the gradient is large. And for symmetric B the two forms agree to exactly 0.0e+00, so the shortcut passes every symmetric test case you would think to write.

2. Both traps on this page come from the same underlying fact. What is it?

   A That matrix multiplication is not commutative B That a quadratic form only sees the symmetric part of its matrix: x-transpose times the antisymmetric part times x is exactly zero for every x, so that part cannot appear in the derivative C That the trace is linear D That finite differences have a floor
   pch.quizShowAnswer

   B — That a quadratic form only sees the symmetric part of its matrix: x-transpose times the antisymmetric part times x is exactly zero for every x, so that part cannot appear in the derivative — Measured: x-transpose B_a x came out between 3.5e-18 and 4.2e-16 for five random x — zero to within rounding. Any expression for the gradient that retains B_a — like 2 x-transpose B, or Eq 5.108 with a non-symmetric W — is wrong by exactly that retained part.

3. The ten identity bars all sit between 2.5e-11 and 2.7e-10. What does that range tell you?

   A That the identities have small errors B Almost nothing about the identities — it is the central difference's own floor at h = 1e-6. What would be informative is a bar at 1e-1, and the symmetry figure has one at 1.07 C That some identities are more accurate than others D That the tolerance should be tightened
   pch.quizShowAnswer

   B — Almost nothing about the identities — it is the central difference's own floor at h = 1e-6. What would be informative is a bar at 1e-1, and the symmetry figure has one at 1.07 — A test whose resolution is 1e-10 can only ever say 'consistent with exact'. The value of the check is that a wrong form lands nine orders away: the two traps on this page measure 1.20 and 2.9e-01.

4. The cost figure shows finite differencing costing exactly n+1 times the closed form. Where does that come from?

   A Empirically, from the plot B From the arithmetic: differencing needs 2n evaluations of a form costing n-squared plus n, over a closed form costing 2 n-squared — which simplifies to n+1 exactly C From the condition number D From the number of function calls alone
   pch.quizShowAnswer

   B — From the arithmetic: differencing needs 2n evaluations of a form costing n-squared plus n, over a closed form costing 2 n-squared — which simplifies to n+1 exactly — So differencing is not a constant factor worse, it is a factor of the DIMENSION worse: 3x at n = 2 and 257x at n = 256. That is why nobody gradient-checks a large model except on a tiny slice.

5. Given that a closed form and reverse-mode autodiff are the same order, what are these ten identities actually for?

   A Speed B Deriving on paper. They need no computation graph, so they are what you use to work out what a gradient IS, and to sanity-check a library on a small case — not to replace the library C Numerical stability D Reducing memory
   pch.quizShowAnswer

   B — Deriving on paper. They need no computation graph, so they are what you use to work out what a gradient IS, and to sanity-check a library on a small case — not to replace the library — 2n-squared against 3n-squared is a constant factor, so hand-derived gradients buy no asymptotic speed and cost a maintenance burden whenever the model changes. The measured lines are parallel on the log-log plot, which is the whole point.

🧪 Try It Yourself

Exercise 1 – All ten identities

pch.exerciseTag pch.exerciseTitle

Exercise 2 – Trap one: why (B + B-transpose)

pch.exerciseTag pch.exerciseTitle

Exercise 3 – Trap two: Equation 5.108’s symmetry clause

pch.exerciseTag pch.exerciseTitle

Exercise 4 – The cost of each route

pch.exerciseTag pch.exerciseTitle

Exercise 5 – Rebuild 5.103 and 5.108 by composition

pch.exerciseTag pch.exerciseTitle

Recall card

• Every identity in §5.5 is a differential — a statement about one nudge — which is why they are usable by hand where §5.4’s tensors are not.
• Ten equations, fewer ideas. 5.99 to 5.102 are structural rules for any f; 5.104 and 5.105 are the same fact; 5.103 is 5.106 composed with 5.102; 5.108 is 5.107 plus the chain rule. Memorise 5.102, 5.107 and 5.108.
• All ten verified against central differences: worst relative error 2.7e-10, which is the test’s floor rather than the identities’.
• Equation 5.99 comes out at exactly 0.0e+00, because both sides are the same numbers rearranged — a check on the algebra, not the analysis.
• Trap one: Equation 5.107 is x-transpose (B + B-transpose), not 2 x-transpose B. The shortcut is off by 1.20 relative on a general B — more than the gradient itself — and exactly right on a symmetric one, so it passes every symmetric test case.
• Trap two: Equation 5.108’s ‘for symmetric W’ is load-bearing. With a non-symmetric W the stated form is off by 2.9e-01 in one test and 1.07 in another, while the general form — minus e-transpose (W + W-transpose) A — stays at 1e-12.
• Both traps are the same fact: a quadratic form only sees the symmetric part of its matrix, because x-transpose times the antisymmetric part times x is zero for every x — measured between 3.5e-18 and 4.2e-16.
• The trace and transpose in 5.99 to 5.102 mean contractions when f takes a matrix argument — the book’s own Remark says the matrix operations are undefined for higher-order tensors.
• Differencing costs exactly n+1 times a closed form — 3x at n = 2, 257x at n = 256 — because it needs 2n evaluations of an n-squared form.
• A closed form and reverse-mode AD are the same order, 2n² against 3n². Identities buy understanding and paper derivations, not speed.

Next: Backpropagation and Automatic Differentiation — the chain rule applied mechanically, and why it costs about what the function costs.