First derivatives (gradients) tell you the slope; second derivatives tell you the curvature — whether a surface bends up like a bowl, down like a dome, or twists like a saddle. Collected into the Hessian matrix, second derivatives distinguish minima from maxima, drive Newton’s method, and explain why high-dimensional optimization is really a story about saddle points.

A real-life example: is this the bottom of the valley?

Gradient descent stops where the gradient is zero — but a zero gradient could be a valley bottom (minimum), a hilltop (maximum), or a mountain pass (saddle). The gradient alone can’t tell them apart. The Hessian can: its curvature signature says which one you’re at. In deep networks, most zero-gradient points turn out to be saddle points, not minima — a fact that shapes how modern optimizers behave.

Second partial derivatives

For $f(x, y)$ we can differentiate twice, in any combination:

For a twice-continuously-differentiable function, the mixed partials are equal (Schwarz’s theorem): $\frac{\partial^2 f}{\partial x\,\partial y} = \frac{\partial^2 f}{\partial y\,\partial x}$ — order doesn’t matter.

The Hessian

Collecting all second-order partials gives the Hessian matrix $H$ — for $f : \mathbb{R}^n \to \mathbb{R}$ it’s the $n \times n$ symmetric matrix

It measures curvature locally. And because it’s symmetric, its eigenvalues are real (spectral theorem) — and their signs classify the critical point:

See curvature classify a point

Watch a surface morph through the three cases. The Hessian’s eigenvalue signs (shown live) determine whether the point at the center is a minimum, maximum, or saddle — a bowl, a dome, or a twist:

The eigenvalues are the curvatures along the principal axes; when they disagree in sign, you have a saddle — flat in some directions but curving up in others.

Newton’s method

The Hessian powers second-order optimization. Where gradient descent uses only the slope, Newton’s method uses curvature to jump more directly toward the optimum:

It converges much faster near a minimum, but computing and inverting the Hessian is expensive ($O(n^3)$), so large-scale ML mostly uses cheaper first-order methods or Hessian approximations (L-BFGS, Adam’s diagonal preconditioning).

  flowchart TD
  F["f(x)"] --> H["Hessian H (2nd derivatives)"]
  H --> E["eigenvalue signs"]
  E -->|"all +"| MIN["minimum (convex bowl)"]
  E -->|"all −"| MAX["maximum (dome)"]
  E -->|"mixed"| SAD["saddle point"]
  H -.-> NEWT["Newton step: x − H⁻¹∇f"]

NumPy

import numpy as np
 
# f(x, y) = x² + 3y²  ->  H = [[2, 0], [0, 6]]  (both positive → minimum)
def hessian_quadratic(a, b):
    return np.array([[a, 0.0], [0.0, b]])
 
def classify(H):
    eig = np.linalg.eigvalsh(H)          # symmetric → real eigenvalues
    if np.all(eig > 0):  return "minimum"
    if np.all(eig < 0):  return "maximum"
    return "saddle point"
 
for a, b, label in [(2, 6, "bowl"), (-2, -6, "dome"), (2, -6, "twist")]:
    H = hessian_quadratic(a, b)
    print(f"H diag ({a},{b}) eigenvalues {np.linalg.eigvalsh(H)} -> {classify(H)}")
 
# mixed partials are equal (Schwarz): build H numerically and check symmetry
print("Hessian symmetric:", np.allclose(hessian_quadratic(2, 6), hessian_quadratic(2, 6).T))

import numpy as np
 
# f(x, y) = x² + 3y²  ->  H = [[2, 0], [0, 6]]  (both positive → minimum)
def hessian_quadratic(a, b):
    return np.array([[a, 0.0], [0.0, b]])
 
def classify(H):
    eig = np.linalg.eigvalsh(H)          # symmetric → real eigenvalues
    if np.all(eig > 0):  return "minimum"
    if np.all(eig < 0):  return "maximum"
    return "saddle point"
 
for a, b, label in [(2, 6, "bowl"), (-2, -6, "dome"), (2, -6, "twist")]:
    H = hessian_quadratic(a, b)
    print(f"H diag ({a},{b}) eigenvalues {np.linalg.eigvalsh(H)} -> {classify(H)}")
 
# mixed partials are equal (Schwarz): build H numerically and check symmetry
print("Hessian symmetric:", np.allclose(hessian_quadratic(2, 6), hessian_quadratic(2, 6).T))

H diag (2,6) eigenvalues [2. 6.] -> minimum
H diag (-2,-6) eigenvalues [-6. -2.] -> maximum
H diag (2,-6) eigenvalues [-6.  2.] -> saddle point
Hessian symmetric: True

H diag (2,6) eigenvalues [2. 6.] -> minimum
H diag (-2,-6) eigenvalues [-6. -2.] -> maximum
H diag (2,-6) eigenvalues [-6.  2.] -> saddle point
Hessian symmetric: True

Why this matters for ML

• Distinguishing minima from saddles: in high dimensions, critical points are overwhelmingly saddle points — the Hessian’s mixed eigenvalues explain why, and why momentum/noise help escape them.
• Newton and quasi-Newton methods (L-BFGS) use curvature for faster convergence when affordable.
• The Laplace approximation fits a Gaussian using the Hessian of the log-posterior at its mode (next page).

🧪 Try It Yourself

Exercise 1 – Build a Hessian

Exercise 2 – Classify a critical point

Exercise 3 – One Newton step

Recap

• Second derivatives measure curvature; mixed partials are equal for smooth functions (Schwarz).
• The Hessian is the symmetric matrix of second partials; its eigenvalue signs classify critical points (all + → min, all − → max, mixed → saddle).
• The Hessian powers Newton’s method ($\mathbf{x} - H^{-1}\nabla f$) and explains why high-dimensional loss landscapes are dominated by saddle points.

Next: approximating any function with polynomials around a point — Linearization and Multivariate Taylor Series.