Real models depend on many variables — thousands or billions of parameters. The derivative generalizes to the gradient: a vector of partial derivatives that points in the direction the function increases fastest. Gradient descent walks downhill against it. This is the single most important object in machine-learning optimization.

A real-life example: climbing a hill in fog

You’re on a hillside in thick fog and want to reach the top. You can’t see the summit, but you can feel the slope under your feet in every direction. The gradient is the compass that points straight uphill — the steepest way up. To go down (minimize a loss), you walk the opposite way. Gradient descent is exactly this: feel the local slope, step downhill, repeat.

Partial derivatives

For a function $f(x_1, \dots, x_n)$ of several variables, a partial derivative measures how $f$ changes as you nudge one variable, holding the others fixed:

Each partial is an ordinary single-variable derivative — just treat the other variables as constants.

The gradient

Collect all partial derivatives into a row vector — the gradient (also the Jacobian of a scalar function):

Its defining property: the gradient points in the direction of steepest ascent, and its negative points in the direction of steepest descent. It’s also always perpendicular to the level curves (contours) of $f$.

See the gradient field

The heatmap is a function $f(x, y)$ (bright = high). At the moving point, the amber arrow is the gradient — watch it always point “uphill” toward brighter regions, perpendicular to the contour it sits on. The red arrow is its negative: the direction gradient descent would step:

The multivariate chain rule

When variables themselves depend on other variables, partials combine. If $f(x_1, x_2)$ with $x_1(t), x_2(t)$:

Written with the gradient as a row vector, the multivariate chain rule becomes a clean matrix multiplication — the reason we define the gradient as a row. This compact form is what lets backpropagation chain layers together without fussing over dimensions.

  flowchart TD
  F["f(x₁, …, xₙ)"] --> P["partial derivatives ∂f/∂xᵢ
(vary one variable at a time)"]
  P --> G["gradient ∇f = [∂f/∂x₁ … ∂f/∂xₙ]"]
  G --> A["points in steepest-ascent direction"]
  G --> B["perpendicular to level curves"]
  A -.-> GD["−∇f drives gradient descent"]

NumPy

import numpy as np
 
# f(x, y) = x² y + x y³   ->   ∇f = [2xy + y³,  x² + 3xy²]
def f(x, y):  return x**2 * y + x * y**3
def grad(x, y): return np.array([2*x*y + y**3, x**2 + 3*x*y**2])
 
# analytic gradient at (1, 2)
print("analytic ∇f(1,2):", grad(1.0, 2.0))     # [12, 13]
 
# numeric gradient via finite differences (per component)
def numeric_grad(f, x, y, h=1e-6):
    dfdx = (f(x+h, y) - f(x, y)) / h
    dfdy = (f(x, y+h) - f(x, y)) / h
    return np.array([dfdx, dfdy])
print("numeric  ∇f(1,2):", np.round(numeric_grad(f, 1.0, 2.0), 4))
 
# NumPy also has np.gradient for sampled data

import numpy as np
 
# f(x, y) = x² y + x y³   ->   ∇f = [2xy + y³,  x² + 3xy²]
def f(x, y):  return x**2 * y + x * y**3
def grad(x, y): return np.array([2*x*y + y**3, x**2 + 3*x*y**2])
 
# analytic gradient at (1, 2)
print("analytic ∇f(1,2):", grad(1.0, 2.0))     # [12, 13]
 
# numeric gradient via finite differences (per component)
def numeric_grad(f, x, y, h=1e-6):
    dfdx = (f(x+h, y) - f(x, y)) / h
    dfdy = (f(x, y+h) - f(x, y)) / h
    return np.array([dfdx, dfdy])
print("numeric  ∇f(1,2):", np.round(numeric_grad(f, 1.0, 2.0), 4))
 
# NumPy also has np.gradient for sampled data

analytic ∇f(1,2): [12. 13.]
numeric  ∇f(1,2): [12.     13.0001]

analytic ∇f(1,2): [12. 13.]
numeric  ∇f(1,2): [12.     13.0001]

Why this matters for ML

• Gradient descent — the workhorse training algorithm — updates parameters as $\boldsymbol\theta \leftarrow \boldsymbol\theta - \eta\,\nabla_{\boldsymbol\theta} L$: step downhill along the negative gradient.
• The gradient’s direction (steepest ascent) is why descent works; its magnitude informs learning-rate and convergence behavior.
• The multivariate chain rule is the mathematical skeleton of backpropagation.

🧪 Try It Yourself

Exercise 1 – Partial derivatives

Exercise 2 – Numeric gradient check

Exercise 3 – One step of gradient descent

Recap

• A partial derivative varies one input at a time; the gradient stacks them into a row vector $\nabla f$.
• The gradient points in the direction of steepest ascent and is perpendicular to level curves; $-\nabla f$ is steepest descent.
• The multivariate chain rule becomes a matrix product (why the gradient is a row) — the basis of backprop.
• Gradient descent ($\boldsymbol\theta \leftarrow \boldsymbol\theta - \eta\nabla L$) is how virtually every model trains.

Next: functions that output vectors, and their derivative — the Jacobian.