Everything in this chapter rests on one idea: the derivative, the instantaneous rate of change of a function. It’s the slope of the tangent line, the limit of a shrinking secant, and — most importantly for us — the thing that tells gradient descent which way to step. We start in one dimension where it’s easiest to see, then generalize to gradients on the next page.

A real-life example: instantaneous speed

Your car’s odometer tells you distance; your speedometer tells you speed — the derivative of distance with respect to time. Average speed over an hour is a crude difference quotient (distance ÷ time). But your speedometer shows speed right now: the limit as the time interval shrinks to zero. That limit is the derivative.

The difference quotient

For a function $y = f(x)$, the difference quotient is the slope of the secant line through two points:

It’s the average rate of change between $x$ and $x + \delta x$. As we shrink $\delta x$ toward zero, the secant becomes the tangent, and its slope is the derivative:

The derivative points in the direction of steepest ascent of $f$ — the single most important sentence in this chapter.

Watch the secant become the tangent

A point rides along the curve. From it, a secant reaches to a second point $h$ away; as $h$ shrinks (the readout ticks down), the secant swings into the tangent line, and its slope settles on the derivative $f'(x)$:

As $h \to 0$ the secant slope closes in on the tangent slope — the definition of the derivative, made visible.

The rules you’ll actually use

Rather than take limits every time, we use rules. With $f' = \frac{df}{dx}$:

The chain rule is the star: it differentiates compositions, and composition is exactly how neural networks are built (layer after layer). Backpropagation is the chain rule applied at scale.

Taylor series: functions as polynomials

Any smooth function can be approximated near a point $x_0$ by a Taylor polynomial built from its derivatives:

As $n \to \infty$ (for analytic functions) this becomes the Taylor series, an exact representation. It’s how calculators compute $\sin$, $\exp$, and $\log$, and how we build local approximations of loss functions in optimization.

  flowchart LR
  DQ["difference quotient
(secant slope)"] -->|"h → 0"| DERIV["derivative f'(x)
(tangent slope)"]
  DERIV --> RULES["product · quotient · chain rules"]
  DERIV --> TAY["Taylor series
Σ f⁽ᵏ⁾(x₀)/k! (x−x₀)ᵏ"]
  RULES -.-> BP["backpropagation"]

NumPy: numeric and symbolic derivatives

import numpy as np
 
# f(x) = x^3 ; f'(x) = 3x^2. Numeric derivative via the difference quotient.
def f(x): return x**3
def numeric_derivative(f, x, h=1e-6):
    return (f(x + h) - f(x)) / h
 
x0 = 2.0
print("numeric f'(2) :", round(numeric_derivative(f, x0), 4))   # ~12
print("exact   f'(2) :", 3 * x0**2)                              # 12
 
# Chain rule check: h(x) = (2x+1)^4 -> h'(x) = 8(2x+1)^3
def h(x): return (2*x + 1)**4
print("numeric h'(1) :", round(numeric_derivative(h, 1.0), 2))   # ~216
print("exact   h'(1) :", 8 * (2*1 + 1)**3)                       # 216

import numpy as np
 
# f(x) = x^3 ; f'(x) = 3x^2. Numeric derivative via the difference quotient.
def f(x): return x**3
def numeric_derivative(f, x, h=1e-6):
    return (f(x + h) - f(x)) / h
 
x0 = 2.0
print("numeric f'(2) :", round(numeric_derivative(f, x0), 4))   # ~12
print("exact   f'(2) :", 3 * x0**2)                              # 12
 
# Chain rule check: h(x) = (2x+1)^4 -> h'(x) = 8(2x+1)^3
def h(x): return (2*x + 1)**4
print("numeric h'(1) :", round(numeric_derivative(h, 1.0), 2))   # ~216
print("exact   h'(1) :", 8 * (2*1 + 1)**3)                       # 216

numeric f'(2) : 12.0
exact   f'(2) : 12.0
numeric h'(1) : 216.0
exact   h'(1) : 216

numeric f'(2) : 12.0
exact   f'(2) : 12.0
numeric h'(1) : 216.0
exact   h'(1) : 216

Why this matters for ML

• Gradient descent steps against the derivative/gradient to minimize loss — the derivative literally tells the model which way to move.
• The chain rule is backpropagation’s engine: it differentiates the deeply-composed function a neural network computes.
• Taylor approximations underlie optimization theory (why gradient/Newton steps work) and the Laplace approximation in Bayesian ML.

🧪 Try It Yourself

Exercise 1 – Numeric derivative

Exercise 2 – Chain rule

Exercise 3 – Taylor approximation of eˣ

Recap

• The derivative is the limit of the difference quotient — the slope of the tangent, the direction of steepest ascent.
• Use the power, sum, product, quotient, and chain rules instead of taking limits by hand.
• The chain rule differentiates compositions — the seed of backpropagation.
• Taylor series approximate any smooth function with polynomials built from its derivatives.

Next: derivatives of functions with many inputs — the gradient.