What gradient descent does

Gradient descent is a method to minimize a cost function.

It works by repeating:

1. compute direction of steepest increase (gradient)
2. step in the opposite direction (downhill)

  flowchart TD
  A[Start with random parameters] --> B[Compute predictions]
  B --> C[Compute cost]
  C --> D[Compute gradient]
  D --> E[Update parameters]
  E --> B

The update rule

For parameter ww:

w := w - α * (∂Cost/∂w)w := w - α * (∂Cost/∂w)

Where:

• αα is the learning rate

Learning rate intuition

• too small → slow training
• too large → overshoot / diverge

  flowchart LR
  S[Small α] --> Slow[Slow but stable]
  L[Large α] --> Boom[May diverge]

Why it matters for regression

Linear regression can be solved analytically, but gradient descent:

• generalizes to many models (logistic regression, neural nets)
• scales to large datasets

Batch Gradient Descent

The version above is called Batch Gradient Descent because it computes the gradient using the entire training set at every single step. That gradient vector — one partial derivative per parameter — is:

∇MSE(θ) = (2/m) · Xᵀ(Xθ − y)∇MSE(θ) = (2/m) · Xᵀ(Xθ − y)

and the update rule just steps downhill by the learning rate ηη:

θ(next step) = θ − η · ∇MSE(θ)θ(next step) = θ − η · ∇MSE(θ)

import numpy as np
 
np.random.seed(42)
X = 2 * np.random.rand(100, 1)
y = 4 + 3 * X + np.random.randn(100, 1)
X_b = np.c_[np.ones((100, 1)), X]
 
eta = 0.1            # learning rate
n_iterations = 1000
m = 100
theta = np.random.randn(2, 1)   # random initialization
 
for iteration in range(n_iterations):
    gradients = 2 / m * X_b.T.dot(X_b.dot(theta) - y)
    theta = theta - eta * gradients
 
print("theta:", theta.ravel())   # should land close to [4, 3]

import numpy as np
 
np.random.seed(42)
X = 2 * np.random.rand(100, 1)
y = 4 + 3 * X + np.random.randn(100, 1)
X_b = np.c_[np.ones((100, 1)), X]
 
eta = 0.1            # learning rate
n_iterations = 1000
m = 100
theta = np.random.randn(2, 1)   # random initialization
 
for iteration in range(n_iterations):
    gradients = 2 / m * X_b.T.dot(X_b.dot(theta) - y)
    theta = theta - eta * gradients
 
print("theta:", theta.ravel())   # should land close to [4, 3]

Because it scans the whole dataset every step, Batch GD is slow on very large training sets — but it scales well with the number of features, so it beats the Normal Equation once you have hundreds of thousands of features.

Stochastic Gradient Descent (SGD)

Instead of the whole batch, Stochastic GD picks one random instance per step and computes the gradient from just that instance. Much faster per step, and it can even train on datasets too large to fit in memory — but the cost bounces around instead of smoothly decreasing, so the final parameters are good, not optimal, unless you gradually shrink the learning rate using a learning schedule:

import numpy as np
 
n_epochs = 50
t0, t1 = 5, 50   # learning schedule hyperparameters
 
def learning_schedule(t):
    return t0 / (t + t1)
 
theta = np.random.randn(2, 1)
 
for epoch in range(n_epochs):
    for i in range(m):
        random_index = np.random.randint(m)
        xi = X_b[random_index:random_index + 1]
        yi = y[random_index:random_index + 1]
        gradients = 2 * xi.T.dot(xi.dot(theta) - yi)
        eta = learning_schedule(epoch * m + i)
        theta = theta - eta * gradients
 
print("theta:", theta.ravel())

import numpy as np
 
n_epochs = 50
t0, t1 = 5, 50   # learning schedule hyperparameters
 
def learning_schedule(t):
    return t0 / (t + t1)
 
theta = np.random.randn(2, 1)
 
for epoch in range(n_epochs):
    for i in range(m):
        random_index = np.random.randint(m)
        xi = X_b[random_index:random_index + 1]
        yi = y[random_index:random_index + 1]
        gradients = 2 * xi.T.dot(xi.dot(theta) - yi)
        eta = learning_schedule(epoch * m + i)
        theta = theta - eta * gradients
 
print("theta:", theta.ravel())

One pass through the training set (mm steps) is called an epoch. With scikit-learn, you don’t need to write this loop yourself:

from sklearn.linear_model import SGDRegressor
import numpy as np
 
sgd_reg = SGDRegressor(max_iter=1000, tol=1e-3, penalty=None, eta0=0.1)
sgd_reg.fit(X, y.ravel())
print(sgd_reg.intercept_, sgd_reg.coef_)

from sklearn.linear_model import SGDRegressor
import numpy as np
 
sgd_reg = SGDRegressor(max_iter=1000, tol=1e-3, penalty=None, eta0=0.1)
sgd_reg.fit(X, y.ravel())
print(sgd_reg.intercept_, sgd_reg.coef_)

Randomness is a double-edged sword: it helps SGD jump out of local minima on irregular cost surfaces, but it also means the algorithm never truly settles at the minimum — it keeps bouncing nearby.

Mini-batch Gradient Descent

Mini-batch GD is the middle ground: at each step it computes the gradient on a small random batch of instances (say, 32 or 64) instead of one instance or the whole set. This gets a speed boost from vectorized matrix operations (especially on GPUs), and its path is less erratic than plain SGD — though, like SGD, it keeps wandering near the minimum rather than stopping exactly on it.

  flowchart TD
  A["Batch GD
whole training set / step"] --> A2["Smooth, slow steps
stops at the minimum"]
  B["Stochastic GD
1 random instance / step"] --> B2["Fast, noisy steps
bounces near minimum"]
  C["Mini-batch GD
small random batch / step"] --> C2["Balanced speed & noise
bounces less than SGD"]

Comparing the three variants

All three Gradient Descent variants end up near the same minimum for Linear Regression (its MSE cost function is convex — one global minimum, no local traps) — they just take very different paths and speeds to get there.

Visualize it: batch vs stochastic vs mini-batch paths

Same starting point, same bowl-shaped cost surface, three very different walks to the bottom: Batch GD (blue) glides in smoothly and stops right at the minimum; Stochastic GD (amber) darts around noisily and never quite settles; Mini-batch GD (green) is in between — faster than Batch, calmer than Stochastic.

Visualize it

Gradient descent treats the loss as a hill and walks downhill. At each step it moves against the slope (the gradient); where the curve is steep it takes big steps, and as it nears the minimum the slope flattens so the steps shrink and it settles in:

Mini-checkpoint

If training is unstable:

• reduce learning rate
• scale features
• check for exploding gradients (in deep learning)

🧪 Try It Yourself

Exercise 1 – One Batch Gradient Descent Step

Exercise 2 – Learning Schedule for Stochastic GD

Exercise 3 – Train with SGDRegressor

Next

Continue to Regularization - Ridge and Lasso Regression — constrain the weights that Gradient Descent finds so the model doesn’t overfit.