This is the page that makes deep learning work. Training a network means computing the gradient of a loss with respect to millions of parameters. Doing that by hand is hopeless, and doing it naively is far too slow. Backpropagation — reverse-mode automatic differentiation — computes the entire gradient in roughly the same cost as evaluating the function once, by applying the chain rule on a computation graph. Every loss.backward()loss.backward() in PyTorch is this algorithm.

A real-life example: PyTorch’s .backward().backward()

When you write loss.backward()loss.backward(), the framework doesn’t do symbolic calculus and it doesn’t use finite differences. It has recorded every operation you performed as a graph, and it walks that graph backward, multiplying local derivatives and accumulating them via the chain rule. The result is the exact gradient for every parameter — computed automatically, in one sweep.

Gradients in a deep network

A network computes a deeply composed function — layer after layer:

To train it we need $\partial L / \partial \boldsymbol\theta_i$ for every layer’s parameters. The chain rule gives these as products of local derivatives — and crucially, the product for layer $i$ reuses the product already computed for layer $i+1$:

That reuse — computing shared factors once and passing them backward — is the entire efficiency of backpropagation.

Forward pass, then backward pass

Autodiff works in two sweeps over the computation graph. The forward pass computes and stores each intermediate value left-to-right. The backward pass starts from the output with gradient 1 and walks right-to-left, multiplying each edge’s local derivative and accumulating. Watch both sweeps on a small graph $y = (2x + 1)^2$ decomposed as $a = 2x,\; b = a + 1,\; y = b^2$:

The backward pass never recomputes a value — it reuses the forward pass’s stored intermediates and just multiplies local derivatives. That’s why one backward sweep costs about the same as one forward evaluation, no matter how many parameters.

Automatic differentiation: forward vs reverse mode

Backprop is a special case of automatic differentiation — evaluating exact derivatives (to machine precision) by applying the chain rule to a program’s elementary operations. For a chain $x \to a \to b \to y$:

Same answer, different multiplication order. Reverse mode propagates gradients backward (output → input) and is cheap when there are many inputs, few outputs — exactly a neural network (millions of parameters, one scalar loss). Forward mode is cheaper the other way around.

  flowchart LR
  X["inputs x"] -->|"forward pass:
compute & store values"| Y["output / loss"]
  Y -->|"backward pass:
chain-rule local derivatives"| X2["gradients ∂L/∂x"]
  X2 -.-> RM["reverse mode = backprop
(many inputs, one loss → cheap)"]

NumPy: a tiny autodiff by hand

import numpy as np
 
# Function: y = (2x + 1)²  via intermediates a=2x, b=a+1, y=b²
x = 1.5
 
# --- forward pass (store intermediates) ---
a = 2 * x
b = a + 1
y = b**2
print("forward: a=%.1f b=%.1f y=%.1f" % (a, b, y))
 
# --- backward pass (reverse-mode chain rule) ---
dy = 1.0                 # seed: ∂y/∂y
db = dy * (2 * b)        # ∂y/∂b = 2b
da = db * 1              # ∂b/∂a = 1
dx = da * 2              # ∂a/∂x = 2
print("backward: ∂y/∂x =", dx)
 
# check against the closed form dy/dx = 4(2x+1)
print("closed form  =", 4 * (2*x + 1))

import numpy as np
 
# Function: y = (2x + 1)²  via intermediates a=2x, b=a+1, y=b²
x = 1.5
 
# --- forward pass (store intermediates) ---
a = 2 * x
b = a + 1
y = b**2
print("forward: a=%.1f b=%.1f y=%.1f" % (a, b, y))
 
# --- backward pass (reverse-mode chain rule) ---
dy = 1.0                 # seed: ∂y/∂y
db = dy * (2 * b)        # ∂y/∂b = 2b
da = db * 1              # ∂b/∂a = 1
dx = da * 2              # ∂a/∂x = 2
print("backward: ∂y/∂x =", dx)
 
# check against the closed form dy/dx = 4(2x+1)
print("closed form  =", 4 * (2*x + 1))

forward: a=3.0 b=4.0 y=16.0
backward: ∂y/∂x = 16.0
closed form  = 16.0

forward: a=3.0 b=4.0 y=16.0
backward: ∂y/∂x = 16.0
closed form  = 16.0

Why this matters for ML

• Backprop is how every neural network trains — it’s the algorithm behind .backward().backward(), GradientTapeGradientTape, and grad()grad().
• Reverse mode’s efficiency (one backward pass ≈ one forward pass) is what makes training billion-parameter models feasible.
• Understanding the computation graph explains memory use (intermediates must be stored), gradient checkpointing, and why some operations aren’t differentiable.

🧪 Try It Yourself

Exercise 1 – Backward pass by hand

Exercise 2 – Reuse a shared factor

Exercise 3 – Reverse mode is cheaper here

Recap

• A network is a composed function; its gradient is a chain-rule product that reuses shared factors — the insight behind backpropagation.
• Automatic differentiation computes exact gradients via a computation graph: a forward pass stores values, a backward pass accumulates gradients.
• Reverse mode = backprop, cheap when inputs ≫ outputs (a network’s millions of params → one loss). One backward pass ≈ one forward evaluation.

Next: second derivatives and curvature — the Hessian.