Vector Calculus — Overview

Chapters 2–4 gave us the objects of machine learning (vectors, matrices, decompositions). Chapter 5 gives us the verb: how a model learns. Learning is optimization, optimization runs on gradients, and gradients are vector calculus. Every time a neural network trains, it is computing derivatives of a loss with respect to millions of parameters — and this chapter is the machinery that makes that possible.

Why gradients are the whole game

A model has knobs (parameters) and a loss that says how wrong it is. Training = turning the knobs to shrink the loss. The gradient points in the direction the loss increases fastest, so stepping against it decreases the loss — that’s gradient descent. The entire deep-learning revolution rests on computing this gradient efficiently, which is exactly what backpropagation (reverse-mode automatic differentiation) does.

The concept map

diagram Diagram mermaid

  flowchart TD
  DQ["Difference quotient"] --> PD["Partial derivatives"]
  PD --> GRAD["Gradient (∇f)"]
  GRAD --> JAC["Jacobian"]
  JAC --> HESS["Hessian (2nd order)"]
  PD --> BP["Backpropagation /
automatic differentiation"]
  HESS --> TS["Taylor series"]
  BP -.-> OPT["Optimization / training (Ch 7)"]
  GRAD -.-> OPT
  JAC -.-> PROB["change of variables (Ch 6)"]
  GRAD -.-> REG["regression, PCA (Ch 9, 10)"]

(Adapted from Figure 5.2 of the book.)

The through-line

Differentiation of Univariate Functions — the derivative as a slope; the rules (product, quotient, chain); Taylor series.
Partial Differentiation and Gradients — derivatives of multivariable functions; the gradient as the direction of steepest ascent.
Gradients of Vector-Valued Functions — the Jacobian: the derivative of a function that outputs a vector.
Gradients of Matrices — derivatives with respect to matrices, which become higher-dimensional tensors.
Useful Identities for Computing Gradients — a cheat sheet of matrix-gradient formulas, plus gradient checking.
Backpropagation and Automatic Differentiation — the chain rule on a computation graph; how deep networks actually get their gradients.
Higher-Order Derivatives — the Hessian: curvature, used to tell minima from saddle points.
Linearization and Multivariate Taylor Series — approximating any function with polynomials around a point.

Where each idea shows up in ML

Vector-calculus idea	Machine learning payoff
Derivative / gradient	Gradient descent — how every model trains
Chain rule	The mathematical core of backpropagation
Jacobian	Normalizing flows, sensitivity, change of variables
Hessian	Newton’s method, saddle-point detection, curvature
Backprop / autodiff	PyTorch/TensorFlow `.backward().backward()` — automatic gradients
Taylor series	Local approximations, Laplace approximation, optimization theory

Prerequisites

High-school calculus helps (derivatives, the chain rule) but we re-derive the essentials.
Chapter 2 — Linear Algebra and Chapter 3 — Analytic Geometry for vectors, matrices, and inner products.

Start with Differentiation of Univariate Functions.

If this helped you, consider buying me a coffee ☕

Buy me a coffee