Deriving matrix gradients from scratch every time is slow and error-prone. In practice you reach for a small set of identities — ready-made formulas for the gradients that appear over and over in machine learning (least squares, log-likelihoods, regularizers). This page collects the ones worth memorizing, and shows how to gradient-check any derivation numerically.

A real-life example: deriving the least-squares gradient

To fit linear regression you minimize $\lVert \mathbf{y} - \Phi\boldsymbol\theta\rVert^2$. Finding the optimum needs its gradient with respect to $\boldsymbol\theta$. Instead of expanding everything by hand, one identity — $\partial(\mathbf{x}^\top B\mathbf{x})/\partial\mathbf{x} = \mathbf{x}^\top(B + B^\top)$ — plus the chain rule gets you there in two lines. These identities are the difference between a page of algebra and a one-liner.

The cheat sheet

Here $\text{tr}$ is the trace, $\det$ the determinant, and $f(X)^{-1}$ the inverse (assumed to exist). $\mathbf{a}, \mathbf{b}$ are constant vectors; $B, W$ constant matrices.

Vector-in identities (the ones you’ll use most):

Matrix-in identities:

The two most-used in everyday ML are $\partial(\mathbf{a}^\top\mathbf{x})/\partial\mathbf{x} = \mathbf{a}^\top$ (linear terms) and $\partial(\mathbf{x}^\top B\mathbf{x})/\partial\mathbf{x} = \mathbf{x}^\top(B + B^\top)$ (quadratic terms). For symmetric $B$ the latter simplifies to $2\mathbf{x}^\top B$.

The safety net: gradient checking

However you derive a gradient, verify it numerically. Compare your analytic gradient to a finite-difference approximation:

But there’s a catch: too-large $h$ has truncation error; too-small $h$ has floating-point round-off error. The total error is a V-shape with a sweet spot in the middle. Watch the error sweep as $h$ shrinks — it drops, bottoms out (~$10^{-8}$ for forward differences), then rises again as round-off takes over:

The practical rule: use $h \approx 10^{-6}$ to $10^{-8}$ for forward differences, and flag a bug if the relative error $\frac{\lVert \text{numeric} - \text{analytic}\rVert}{\lVert \text{numeric} + \text{analytic}\rVert}$ exceeds $\sim 10^{-6}$.

  flowchart LR
  D["derive gradient
(by identity or chain rule)"] --> C["gradient check
vs finite differences"]
  C -->|"match"| OK["ship it"]
  C -->|"mismatch"| BUG["bug in derivation / code"]

NumPy: using identities and gradient checking

import numpy as np
 
# Identity: ∂(xᵀ B x)/∂x = xᵀ(B + Bᵀ);  for symmetric B this is 2 xᵀ B
B = np.array([[2.0, 1.0], [1.0, 3.0]])       # symmetric
x = np.array([1.0, 2.0])
 
analytic = x @ (B + B.T)                       # = 2 xᵀB here
print("analytic gradient:", analytic)
 
# Gradient check with finite differences
def f(v): return v @ B @ v
def numeric_grad(f, x, h=1e-6):
    g = np.zeros_like(x)
    for i in range(len(x)):
        xp = x.copy(); xp[i] += h
        g[i] = (f(xp) - f(x)) / h
    return g
 
num = numeric_grad(f, x)
print("numeric  gradient:", np.round(num, 4))
rel = np.linalg.norm(num - analytic) / np.linalg.norm(num + analytic)
print("relative error:", f"{rel:.2e}", "-> OK" if rel < 1e-4 else "-> BUG")

import numpy as np
 
# Identity: ∂(xᵀ B x)/∂x = xᵀ(B + Bᵀ);  for symmetric B this is 2 xᵀ B
B = np.array([[2.0, 1.0], [1.0, 3.0]])       # symmetric
x = np.array([1.0, 2.0])
 
analytic = x @ (B + B.T)                       # = 2 xᵀB here
print("analytic gradient:", analytic)
 
# Gradient check with finite differences
def f(v): return v @ B @ v
def numeric_grad(f, x, h=1e-6):
    g = np.zeros_like(x)
    for i in range(len(x)):
        xp = x.copy(); xp[i] += h
        g[i] = (f(xp) - f(x)) / h
    return g
 
num = numeric_grad(f, x)
print("numeric  gradient:", np.round(num, 4))
rel = np.linalg.norm(num - analytic) / np.linalg.norm(num + analytic)
print("relative error:", f"{rel:.2e}", "-> OK" if rel < 1e-4 else "-> BUG")

analytic gradient: [ 8. 14.]
numeric  gradient: [ 8.     14.0001]
relative error: 3.02e-06 -> OK

analytic gradient: [ 8. 14.]
numeric  gradient: [ 8.     14.0001]
relative error: 3.02e-06 -> OK

Why this matters for ML

• Deriving loss gradients (regression, logistic regression, Gaussian likelihoods) is fast with these identities instead of index-by-index calculus.
• Gradient checking is the standard sanity test when implementing a custom layer or loss — it catches sign errors and transpose mistakes instantly.
• Understanding the identities helps you read papers where gradients are stated without derivation.

🧪 Try It Yourself

Exercise 1 – Linear-term identity

Exercise 2 – Quadratic-form identity

Exercise 3 – Gradient check a derivation

Recap

• A handful of identities ($\mathbf{a}^\top\mathbf{x}$, $\mathbf{x}^\top B\mathbf{x}$, trace, determinant, inverse) cover most ML gradient derivations.
• Always gradient-check against finite differences; watch for the truncation/round-off trade-off (best $h \approx 10^{-6}$–$10^{-8}$).
• These tools turn page-long derivations into one-liners and catch bugs instantly.

Next: how frameworks compute these gradients automatically at scale — Backpropagation and Automatic Differentiation.