Vector Calculus Overview

Chapters 2 to 4 gave you the objects — vectors, matrices, decompositions. Chapter 5 gives you the verb. The book opens by naming three problems it is for:

• linear regression (Chapter 9), where you optimise weights to maximise a likelihood;
• neural-network autoencoders, where you minimise a reconstruction error “by repeated application of the chain rule”;
• Gaussian mixture models (Chapter 11), where you optimise the location and shape of each component.

All three are optimisation, all three run on gradients, and the whole chapter is about computing gradients of things that are not scalars.

The one thing to take from this chapter

Not a formula — a rule about shapes:

The derivative’s shape is the output’s shape followed by the input’s shape.

$f$ maps	derivative	shape	book
$\mathbb{R} \to \mathbb{R}$	derivative	scalar	Def 5.2
$\mathbb{R}^n \to \mathbb{R}$	gradient	$1 \times n$ (a row)	Eq 5.40
$\mathbb{R}^n \to \mathbb{R}^m$	Jacobian	$m \times n$	Eq 5.56
$\mathbb{R}^n \to \mathbb{R}$, twice	Hessian	$n \times n$	Eq 5.147
$\mathbb{R}^n \to \mathbb{R}^m$, twice	tensor	$m \times n \times n$	§5.7 Remark
$\mathbb{R}^{p\times q} \to \mathbb{R}^{m\times n}$	tensor	$(m\times n)\times(p\times q)$	§5.4

Every dimension error in the chapter is a violation of that one line, and if you can state the shape of every factor before computing any of them, the rest is bookkeeping. That is exactly why the book’s Exercises 5.7 and 5.8 ask for the dimensions explicitly rather than just the answers.

And there is really only one rule: Equation 5.32’s chain rule. §5.2, §5.3, §5.6 and §5.8 are all that same rule, at increasing levels of generality, always read as a left-to-right matrix product.

The chapter’s own map

The book’s Figure 5.2, as a diagram:

diagram Diagram mermaid

  flowchart TD
  DQ["Difference quotient
Def 5.1, Def 5.2"] -->|"defines"| PD["Partial derivatives
Def 5.5"]
  PD -->|"collected in"| GRAD["Gradient, 1 x n
Eq 5.40"]
  GRAD --> JAC["Jacobian, m x n
Eq 5.56"]
  PD -->|"collected in"| HESS["Hessian, n x n
Eq 5.147"]
  GRAD --> TS["Taylor series
Def 5.7, Def 5.8"]
  HESS --> TS
  JAC --> BP["Backpropagation
Eq 5.143-5.145"]
  BP -->|"used in"| C7["Ch 7 Optimization"]
  HESS -->|"used in"| C7
  JAC -->|"used in"| C6["Ch 6 Probability
change of variables"]
  GRAD -->|"used in"| C9["Ch 9 Regression"]
  GRAD -->|"used in"| C10["Ch 10 Dimensionality reduction"]
  TS -->|"used in"| C11["Ch 11 Density estimation"]
  JAC -->|"used in"| C12["Ch 12 Classification"]

The through-line, and what each page settles

#	Page	The question it answers
5.1	Differentiation of Univariate Functions	What is a derivative, where does the difference quotient stop working in floating point, and what does a Taylor series actually promise?
5.2	Partial Differentiation and Gradients	Why is the gradient a row, and are the two things everyone says about it — steepest ascent, perpendicular to the level set — actually true?
5.3	Gradients of Vector-Valued Functions	What does the Jacobian’s determinant measure, and where does that measurement break?
5.4	Gradients of Matrices	What shape is the derivative of a matrix with respect to a matrix, and how do you organise it?
5.5	Useful Identities for Computing Gradients	Ten identities, each verified — and the two whose conditions are routinely dropped.
5.6	Backpropagation and Automatic Differentiation	Why does the derivative cost about what the function costs, and why is reverse mode the only option for a scalar loss?
5.7	Higher-Order Derivatives	What does “the Hessian measures curvature” mean as an equation, and when does the second-order test say nothing at all?
5.8	Linearization and Multivariate Taylor Series	How are the gradient and the Hessian the first two terms of one series, and why does practice stop at the second?
—	Chapter 5 Exercises and Solutions	All nine of the book’s exercises, worked and checked against finite differences.
—	Chapter 5 Formula Sheet	Every definition, identity and measured number on one page.

Two reading paths

If you are here for backpropagation (the shortest honest route): §5.1’s chain rule → §5.2’s gradient-as-a-row → §5.3’s Jacobian → §5.6. You can skip §5.4, §5.5, §5.7 and §5.8 and come back.

If you are here for optimisation (Chapter 7’s prerequisites): §5.1 → §5.2 → §5.7’s Hessian → §5.8’s Taylor series. §5.7 and §5.8 are what Newton’s method, trust regions and the Laplace approximation are built from.

Either way, do §5.5’s identities when you first need a closed-form gradient, not before.

What the chapter looks like once it has been measured

Every number below is measured on the page it belongs to, not quoted from anywhere.

Claim	Measured
“Let $h \to 0$”	central differences improve to about $h = 10^{-6}$, then get worse — round-off beats truncation
The gradient is the steepest direction	swept over 360 directions, nothing beats $\lVert\nabla f\rVert$, and the exact gradient direction attains it to $10^{-12}$
The gradient is perpendicular to the level set	worst normalised deviation across a whole field: $10^{-12}$
$\det\mathbf{J}$ is the local area factor	first-order convergent — away from where $\det\mathbf{J}$ changes sign, where a signed area is not what you measured
“The derivative costs what the function costs”	6 forward operations against 16 in the reverse sweep: $2.67\times$, a constant
Reverse mode beats forward mode	crossover at $n = 3$ for a scalar loss; a network has millions of parameters
AD versus finite differences	$2.2\times10^{-16}$ against the best of 200 central differences at $9.8\times10^{-12}$
Mixed partials commute	only for twice continuously differentiable $f$: the standard counterexample’s two iterated limits differ by exactly $2$
Eigenvalue signs classify a critical point	except when one is zero — and three functions share one singular Hessian with three different answers
Newton’s method is better	8 steps against gradient descent’s 426, and it lands exactly on the saddle of $x_1^2 - x_2^2$
Higher-order Taylor is better	as $r \to 0$: fitted orders $1.067$, $2.005$, $3.019$. At $r = 3.2$ the quadratic model was five times worse than the constant
Why nobody goes past second order	$5.0\times10^{11}$ distinct Hessian entries at a million parameters — 4 TB in float32

An orientation tool

Before reading anything, get the shape rule into your hands:

sketch Which derivative do I need? p5.js Pause Restart

Pick what your function takes in and what it puts out, and how many times you are differentiating. The panel names the object, its shape, the book's equation number, and the page that covers it. This is the shape rule made clickable — the single most useful thing to internalise before starting the chapter.

Prerequisites, and the bridge if you need one

You need	From
vectors, matrices, matrix products, shapes	Chapter 2 — Linear Algebra
inner products, norms, orthogonality	Chapter 3 — Analytic Geometry
symmetric matrices have real eigenvalues and an orthonormal eigenbasis	Chapter 4 — Eigenvalues and Eigenvectors
positive definiteness, and Cholesky as a cheap test for it	Chapter 4 — Cholesky Decomposition

School calculus helps but is not assumed — §5.1 rebuilds the derivative from Definition 5.1’s difference quotient. The one genuine dependency is Chapter 4’s spectral theorem: §5.7’s entire classification rests on the Hessian being symmetric, which is what makes its eigenvalues real.

On notation, once

The book writes a function as $f : \mathbb{R}^D \to \mathbb{R}$, $\mathbf{x} \mapsto f(\mathbf{x})$ (Equations 5.1a and 5.1b) — the first line gives the shapes, the second the rule. Example 5.1 does it for the dot product: $f : \mathbb{R}^2 \to \mathbb{R}$, $\mathbf{x} \mapsto x_1^2 + x_2^2$. Getting into the habit of writing the shapes line first is most of what makes this chapter easy.

Where each section is used later

Section	Used in
§5.1 chain rule	everywhere; §5.6 is this applied mechanically
§5.2 gradient as a row	Chapter 7’s optimality conditions, Chapter 9’s normal equations
§5.3 Jacobian	Chapter 6’s change of variables, normalising flows
§5.4 matrix gradients	Chapter 9’s linear regression, Chapter 10’s PCA
§5.5 identities	anywhere a closed-form gradient beats differentiating
§5.6 backpropagation	every neural network ever trained
§5.7 Hessian	Chapter 7’s Newton and convexity, the Laplace approximation
§5.8 Taylor	Chapter 7’s trust regions, Gauss–Newton, the Laplace approximation

pch.quizTag Before you start

1. What shape is the derivative of a function from R^n to R^m?

   A n by m B m by n — the output's shape followed by the input's shape, which is the rule the whole chapter runs on C 1 by n D It depends on the function
   pch.quizShowAnswer

   B — m by n — the output's shape followed by the input's shape, which is the rule the whole chapter runs on — The Jacobian, Equation 5.56. And the same rule gives the gradient as 1 by n, the Hessian as n by n, and a vector field's second derivative as m by n by n.

2. Why does the book insist the gradient is a ROW vector?

   A Because gradients are covectors B So the chain rule is a plain left-to-right matrix product with the shapes lining up, and so the convention still works when the output becomes a vector C To match physics notation D It is arbitrary
   pch.quizShowAnswer

   B — So the chain rule is a plain left-to-right matrix product with the shapes lining up, and so the convention still works when the output becomes a vector — Equation 5.40 gives both reasons. The payoff shows up immediately at Equation 5.148, where grad f times (x - x0) needs no transpose, and again at every chain rule in the chapter.

3. How many genuinely different rules does this chapter contain?

   A One per section B Essentially one: Equation 5.32's chain rule, applied at four levels of generality — scalar, vector input, vector output, and tensors C Two: the chain rule and the product rule D Ten, the identities in Section 5.5
   pch.quizShowAnswer

   B — Essentially one: Equation 5.32's chain rule, applied at four levels of generality — scalar, vector input, vector output, and tensors — Section 5.5's identities are consequences, not new rules, and Section 5.6 is the chain rule applied mechanically to a graph. Recognising that the chapter is one rule at four scales is what stops it feeling like a list to memorise.

4. Which prerequisite does Section 5.7 genuinely depend on?

   A Chapter 3's inner products B Chapter 4's spectral theorem — the classification of critical points rests entirely on the Hessian being symmetric, which is what makes its eigenvalues real and its eigenvectors orthogonal C Chapter 2's Gaussian elimination D Chapter 4's SVD
   pch.quizShowAnswer

   B — Chapter 4's spectral theorem — the classification of critical points rests entirely on the Hessian being symmetric, which is what makes its eigenvalues real and its eigenvectors orthogonal — Equation 5.146 gives the symmetry, Chapter 4 turns symmetry into real eigenvalues, and the eigenvalue SIGNS are the whole second-order test. Take away the symmetry and the test does not exist.

Recall card

• The chapter is one rule at four scales. Equation 5.32’s chain rule, applied to a scalar, to a vector input, to a vector output, and to tensors — always as a left-to-right matrix product.
• The shape rule: the derivative’s shape is the output’s shape followed by the input’s shape. Scalar of a vector is 1 by n, vector of a vector is m by n, twice-differentiated scalar of a vector is n by n, twice-differentiated vector field is m by n by n.
• Equation 5.40 makes the gradient a ROW for two reasons the book gives: the chain rule becomes a plain matrix product, and the convention survives when the output becomes a vector.
• State the shapes before computing anything. Every dimension error in the chapter violates the shape rule, which is why the book’s Exercises 5.7 and 5.8 ask for dimensions explicitly.
• Figure 5.2’s map: the difference quotient defines partial derivatives, which are collected into the gradient, the Jacobian and the Hessian, which feed the Taylor series — and out into Chapters 6, 7, 9, 10, 11 and 12.
• The Hessian is the last object in the chapter with a name. One more index and everything is “a tensor”, which is precisely where Section 5.8 stops being practical.
• Section 5.7 depends on Chapter 4’s spectral theorem. Equation 5.146 gives symmetry, symmetry gives real eigenvalues, and the eigenvalue signs are the entire second-order test.
• Two reading paths. For backpropagation: 5.1, 5.2, 5.3, 5.6. For optimisation: 5.1, 5.2, 5.7, 5.8. Do Section 5.5’s identities when you first need a closed-form gradient.

Start with: Differentiation of Univariate Functions — the derivative from Definition 5.1, and the point where “let $h \to 0$” stops being true in floating point.