Skip to content
Python Central Hub

Regularization - Ridge and Lasso Regression

Why regularization exists

Regression models can overfit when:

  • there are many features
  • features are noisy
  • polynomial degree is high

Overfitting often shows up as:

  • great training error
  • bad validation/test error

Regularization adds a penalty that discourages overly complex solutions.

Ridge Regression (L2)

Ridge (also called Tikhonov regularization) minimizes:

J(θ) = MSE(θ) + λ · (1/2) · Σ(θi²)J(θ) = MSE(θ) + λ · (1/2) · Σ(θi²) for i = 1..n

Note the sum starts at i = 1i = 1, not 00 — the bias term θ0θ0 is never regularized. Effect:

  • shrinks coefficients toward 0
  • usually keeps all features (rarely exactly 0)
  • if λ = 0λ = 0, Ridge is just plain Linear Regression; if λλ is huge, all weights shrink close to 0 and the model becomes a flat line at the mean of yy

Ridge also has a closed-form solution, a small variant of the Normal Equation:

θ̂ = (XᵀX + λA)⁻¹ Xᵀyθ̂ = (XᵀX + λA)⁻¹ Xᵀy

where AA is the identity matrix with a 00 in the top-left corner (so the bias term is excluded).

Lasso Regression (L1)

Lasso (Least Absolute Shrinkage and Selection Operator) minimizes:

J(θ) = MSE(θ) + λ · Σ|θi|J(θ) = MSE(θ) + λ · Σ|θi| for i = 1..n

Effect:

  • can push some coefficients exactly to 0
  • performs automatic feature selection, producing a sparse model

Why does Lasso zero out weights while Ridge only shrinks them? The book’s geometric intuition: the L1 penalty’s contour lines have sharp corners on the axes, so Gradient Descent tends to get pushed exactly onto an axis (where one weight is 0) before rolling down the remaining “gutter.” Ridge’s L2 penalty has smooth circular contours, so weights shrink continuously but rarely hit exactly 0.

Elastic Net

Elastic Net is a mix of both penalties, controlled by a mix ratio rr (scikit-learn calls it l1_ratiol1_ratio):

J(θ) = MSE(θ) + r·λ · Σ|θi| + (1 − r)/2 · λ · Σ(θi²)J(θ) = MSE(θ) + r·λ · Σ|θi| + (1 − r)/2 · λ · Σ(θi²)

  • r = 0r = 0 → pure Ridge
  • r = 1r = 1 → pure Lasso

The book’s rule of thumb: almost always regularize at least a little. Ridge is a good default; if you suspect only a few features actually matter, prefer Lasso or Elastic Net. Elastic Net is usually safer than pure Lasso when you have more features than instances, or when features are strongly correlated.

diagram Diagram mermaid

Scikit-learn examples

Ridge, Lasso, and Elastic Net
from sklearn.linear_model import Ridge, Lasso, ElasticNet
import numpy as np
 
np.random.seed(42)
X = 3 * np.random.rand(50, 1)
y = 1 + 0.5 * X + np.random.randn(50, 1)
 
ridge = Ridge(alpha=1.0, solver="cholesky")   # alpha is λ
ridge.fit(X, y)
print("Ridge:", ridge.predict([[1.5]]))
 
lasso = Lasso(alpha=0.1)
lasso.fit(X, y)
print("Lasso:", lasso.predict([[1.5]]))
 
elastic_net = ElasticNet(alpha=0.1, l1_ratio=0.5)   # l1_ratio is r
elastic_net.fit(X, y)
print("Elastic Net:", elastic_net.predict([[1.5]]))
Ridge, Lasso, and Elastic Net
from sklearn.linear_model import Ridge, Lasso, ElasticNet
import numpy as np
 
np.random.seed(42)
X = 3 * np.random.rand(50, 1)
y = 1 + 0.5 * X + np.random.randn(50, 1)
 
ridge = Ridge(alpha=1.0, solver="cholesky")   # alpha is λ
ridge.fit(X, y)
print("Ridge:", ridge.predict([[1.5]]))
 
lasso = Lasso(alpha=0.1)
lasso.fit(X, y)
print("Lasso:", lasso.predict([[1.5]]))
 
elastic_net = ElasticNet(alpha=0.1, l1_ratio=0.5)   # l1_ratio is r
elastic_net.fit(X, y)
print("Elastic Net:", elastic_net.predict([[1.5]]))

You can also regularize with Gradient Descent via SGDRegressor(penalty="l2")SGDRegressor(penalty="l2") or penalty="l1"penalty="l1" — the penaltypenalty hyperparameter adds the corresponding term straight into the cost function being minimized.

Early stopping

A very different regularization trick for any iterative algorithm: stop training the moment the validation error stops improving and starts climbing back up — that’s the point where the model begins to overfit.

Early stopping (book recipe)
from sklearn.base import clone
from sklearn.linear_model import SGDRegressor
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
import numpy as np
 
np.random.seed(42)
m = 100
X = 6 * np.random.rand(m, 1) - 3
y = 2 + X + 0.5 * X**2 + np.random.randn(m, 1)
X_train, X_val, y_train, y_val = train_test_split(X, y.ravel(), test_size=0.2, random_state=42)
 
poly_scaler = Pipeline([
    ("poly_features", PolynomialFeatures(degree=90, include_bias=False)),
    ("std_scaler", StandardScaler()),
])
X_train_poly_scaled = poly_scaler.fit_transform(X_train)
X_val_poly_scaled = poly_scaler.transform(X_val)
 
sgd_reg = SGDRegressor(max_iter=1, tol=-np.inf, warm_start=True,
                        penalty=None, learning_rate="constant", eta0=0.0005)
 
minimum_val_error = float("inf")
best_epoch, best_model = None, None
 
for epoch in range(200):
    sgd_reg.fit(X_train_poly_scaled, y_train)   # continues where it left off
    y_val_predict = sgd_reg.predict(X_val_poly_scaled)
    val_error = mean_squared_error(y_val, y_val_predict)
    if val_error < minimum_val_error:
        minimum_val_error = val_error
        best_epoch = epoch
        best_model = clone(sgd_reg)
 
print("best epoch:", best_epoch, "min val error:", round(minimum_val_error, 3))
Early stopping (book recipe)
from sklearn.base import clone
from sklearn.linear_model import SGDRegressor
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
import numpy as np
 
np.random.seed(42)
m = 100
X = 6 * np.random.rand(m, 1) - 3
y = 2 + X + 0.5 * X**2 + np.random.randn(m, 1)
X_train, X_val, y_train, y_val = train_test_split(X, y.ravel(), test_size=0.2, random_state=42)
 
poly_scaler = Pipeline([
    ("poly_features", PolynomialFeatures(degree=90, include_bias=False)),
    ("std_scaler", StandardScaler()),
])
X_train_poly_scaled = poly_scaler.fit_transform(X_train)
X_val_poly_scaled = poly_scaler.transform(X_val)
 
sgd_reg = SGDRegressor(max_iter=1, tol=-np.inf, warm_start=True,
                        penalty=None, learning_rate="constant", eta0=0.0005)
 
minimum_val_error = float("inf")
best_epoch, best_model = None, None
 
for epoch in range(200):
    sgd_reg.fit(X_train_poly_scaled, y_train)   # continues where it left off
    y_val_predict = sgd_reg.predict(X_val_poly_scaled)
    val_error = mean_squared_error(y_val, y_val_predict)
    if val_error < minimum_val_error:
        minimum_val_error = val_error
        best_epoch = epoch
        best_model = clone(sgd_reg)
 
print("best epoch:", best_epoch, "min val error:", round(minimum_val_error, 3))

warm_start=Truewarm_start=True means each call to .fit().fit() picks up training where the last one left off, instead of restarting from scratch — essential for checking the validation error one epoch at a time.

Important: scale features

Regularization is sensitive to feature scale.

Use StandardScalerStandardScaler in a pipeline.

Choosing λ (alpha)

  • use validation or cross-validation
  • RidgeCVRidgeCV / LassoCVLassoCV can help

Visualize it

Watch what happens to a set of feature weights as you dial up the regularization strength λλ: Ridge (blue bars) shrinks every weight smoothly toward zero, while Lasso (amber bars) drives the smaller, less important weights all the way to exactly zero — feature selection in action.

sketch Ridge vs Lasso shrinking coefficients p5.js
As λ increases, Ridge shrinks all weights smoothly; Lasso pushes the smaller weights to exactly zero.

Mini-checkpoint

If you have 1000 features:

  • which regularization might help reduce features automatically?

(Usually Lasso.)

🧪 Try It Yourself

Exercise 1 – Fit a Ridge Model

Exercise 2 – Watch Lasso Zero Out a Weight

Exercise 3 – Mix Ridge and Lasso with Elastic Net

Next

Continue to Metrics - R-Squared and Adjusted R-Squared — learn how to score any of these regularized (or plain) models once they’re trained.

If this helped you, consider buying me a coffee ☕

Buy me a coffee

Was this page helpful?

Let us know how we did