Linearization and Multivariate Taylor Series

Complicated functions are hard; polynomials are easy. The Taylor series lets you replace any smooth function, near a point, with a polynomial built from its derivatives — starting with a flat constant, then a tangent line (linearization), then a parabola, and so on. Each extra term hugs the function better. This local-approximation trick underlies optimization theory, the extended Kalman filter, and the Laplace approximation in Bayesian ML.

A real-life example: your GPS trusts a straight line

Over a few meters, the curved surface of the Earth is indistinguishable from a flat plane — so local navigation, AR overlays, and physics engines happily use a linear (first-order) approximation. That’s linearization: replacing a nonlinear function by its tangent near the point of interest. It’s wrong far away, but locally it’s accurate and vastly cheaper to compute.

Linearization: the first-order Taylor approximation

The gradient gives a locally linear approximation of $f$ around a point $\mathbf{x}_0$ :

f(\mathbf{x}) \approx f(\mathbf{x}_0) + (\nabla_{\mathbf{x}} f)(\mathbf{x}_0)(\mathbf{x} - \mathbf{x}_0).

This is the tangent line (or tangent plane). It’s exact at $\mathbf{x}_0$ and degrades as you move away — which is fine as long as you stay local.

Watch higher orders hug the curve

Here’s the univariate picture. The black curve is $f(x) = \sin(x) + \cos(x)$ . The colored dashed curves are Taylor polynomials of increasing degree around $x_0 = 0$ : $T_0$ is a flat constant, $T_1$ the tangent line, $T_2$ a parabola, $T_3$ a cubic — each matching more of the curve before peeling away. Watch the order climb and the fit tighten:

sketch Taylor polynomials approximating a curve p5.js

f(x) = sin(x) + cos(x) (white) with Taylor polynomials T₀…T₄ around x₀ = 0. Each higher-order polynomial matches the function over a wider interval before diverging.

By $T_4$ the polynomial tracks $\sin + \cos$ across a wide interval — a good enough stand-in for the real function.

The multivariate Taylor series

For $f : \mathbb{R}^D \to \mathbb{R}$ smooth at $\mathbf{x}_0$ , with difference vector $\boldsymbol\delta = \mathbf{x} - \mathbf{x}_0$ :

f(\mathbf{x}) = \sum_{k=0}^{\infty} \frac{D^k_{\mathbf{x}} f(\mathbf{x}_0)}{k!}\,\boldsymbol\delta^k,

where $D^k_{\mathbf{x}} f$ is the $k$ -th total derivative. The first three terms are the ones you’ll actually use:

\underbrace{f(\mathbf{x}_0)}_{k=0} + \underbrace{\nabla f(\mathbf{x}_0)\,\boldsymbol\delta}_{k=1,\ \text{gradient}} + \underbrace{\tfrac{1}{2}\boldsymbol\delta^\top H(\mathbf{x}_0)\,\boldsymbol\delta}_{k=2,\ \text{Hessian}} + \cdots

The gradient supplies the linear term and the Hessian the quadratic term. Higher-order terms use $k$ -fold outer products $\boldsymbol\delta^k$ , which become tensors — rarely needed in practice, since first- and second-order approximations dominate ML.

diagram Diagram mermaid

  flowchart LR
  F["f(x) near x₀"] --> T0["T₀: constant f(x₀)"]
  T0 --> T1["T₁: + ∇f·δ  (linearization / tangent)"]
  T1 --> T2["T₂: + ½ δᵀHδ  (quadratic / curvature)"]
  T2 --> TN["Tₙ: higher-order tensors"]
  T1 -.-> KAL["extended Kalman filter"]
  T2 -.-> LAP["Laplace approximation"]

NumPy

taylor.py

import numpy as np
 
# Univariate: Taylor of sin+cos at 0. f^(k)(0) cycles 1, 1, -1, -1, ...
from math import factorial
coef = [1, 1, -1, -1, 1, 1, -1, -1]
def Tn(x, n):
    return sum(coef[k] / factorial(k) * x**k for k in range(n + 1))
 
x = 0.6
for n in [1, 3, 5]:
    print(f"T{n}({x}) = {Tn(x, n):.5f}   true = {np.sin(x)+np.cos(x):.5f}")
 
# Multivariate quadratic approximation: f(x) ≈ f0 + g·δ + ½ δᵀ H δ
def f(v): x, y = v; return x**2 + x*y + y**2
x0 = np.array([1.0, 1.0])
g  = np.array([2*x0[0] + x0[1], x0[0] + 2*x0[1]])   # gradient at x0
H  = np.array([[2.0, 1.0], [1.0, 2.0]])             # Hessian (constant here)
delta = np.array([0.1, -0.2])
approx = f(x0) + g @ delta + 0.5 * delta @ H @ delta
print("quadratic Taylor:", round(approx, 5), " exact:", round(f(x0 + delta), 5))

taylor.py

import numpy as np
 
# Univariate: Taylor of sin+cos at 0. f^(k)(0) cycles 1, 1, -1, -1, ...
from math import factorial
coef = [1, 1, -1, -1, 1, 1, -1, -1]
def Tn(x, n):
    return sum(coef[k] / factorial(k) * x**k for k in range(n + 1))
 
x = 0.6
for n in [1, 3, 5]:
    print(f"T{n}({x}) = {Tn(x, n):.5f}   true = {np.sin(x)+np.cos(x):.5f}")
 
# Multivariate quadratic approximation: f(x) ≈ f0 + g·δ + ½ δᵀ H δ
def f(v): x, y = v; return x**2 + x*y + y**2
x0 = np.array([1.0, 1.0])
g  = np.array([2*x0[0] + x0[1], x0[0] + 2*x0[1]])   # gradient at x0
H  = np.array([[2.0, 1.0], [1.0, 2.0]])             # Hessian (constant here)
delta = np.array([0.1, -0.2])
approx = f(x0) + g @ delta + 0.5 * delta @ H @ delta
print("quadratic Taylor:", round(approx, 5), " exact:", round(f(x0 + delta), 5))

text

T1(0.6) = 1.60000   true = 1.38014
T3(0.6) = 1.38400   true = 1.38014
T5(0.6) = 1.38012   true = 1.38014
quadratic Taylor: 1.33000  exact: 1.33000

text

T1(0.6) = 1.60000   true = 1.38014
T3(0.6) = 1.38400   true = 1.38014
T5(0.6) = 1.38012   true = 1.38014
quadratic Taylor: 1.33000  exact: 1.33000

(The quadratic Taylor expansion of a quadratic function is exact — no higher terms exist.)

Mathematics for Machine Learning (Section 5.8) presents linearization $f(\mathbf{x}) \approx f(\mathbf{x}_0) + (\nabla_{\mathbf{x}}f)(\mathbf{x}_0)(\mathbf{x} - \mathbf{x}_0)$ (Equation 5.148, Figure 5.12) as the first two terms of the multivariate Taylor series (Definition 5.7). It writes the first terms explicitly — $k=0$ constant, $k=1$ the gradient term $\nabla f\,\boldsymbol \delta$ , $k=2$ the Hessian term $\tfrac{1}{2}\boldsymbol\delta^\top H\boldsymbol\delta$ (Equations 5.156–5.159) — using $k$ -fold outer products $\boldsymbol\delta^k$ that form tensors (Figure 5.13). Example 5.15 expands $f(x,y) = x^2 + 2xy + y^3$ and recovers the polynomial exactly. It notes Taylor expansions power the extended Kalman filter and the Laplace approximation.

Why this matters for ML

Optimization theory: gradient descent is justified by the first-order Taylor term; Newton’s method by the second-order (Hessian) term.
The Laplace approximation fits a Gaussian to a posterior using its second-order Taylor expansion (gradient zero at the mode, Hessian sets the width).
Extended Kalman filters linearize nonlinear dynamics with a first-order Taylor expansion to keep Gaussian state estimates tractable.

🧪 Try It Yourself

Exercise 1 – Linearize a function

Exercise 2 – Taylor polynomial of eˣ

Exercise 3 – Quadratic (Hessian) approximation

Recap

Linearization = the first-order Taylor term: replace $f$ by its tangent near $\mathbf{x}_0$ .
The multivariate Taylor series builds a polynomial from the gradient (linear term), Hessian (quadratic term), and higher-order tensors.
Higher degree → better fit over a wider region; quadratics are captured exactly at second order.
Powers optimization theory, the Laplace approximation, and the extended Kalman filter.

🎓 Chapter 5 complete

You’ve built the calculus of learning: derivatives → gradients → Jacobians → matrix/tensor gradients → identities → backpropagation → Hessians → Taylor series. Together these are exactly what happens inside loss.backward()loss.backward() and every optimizer step.

Head back to the Vector Calculus Overview, or revisit Chapter 4 — Matrix Decompositions. Next in the book: Probability and Distributions — the math of uncertainty.

If this helped you, consider buying me a coffee ☕

Buy me a coffee