What you’ll learn

• why shrinking coefficients toward zero can improve predictions — the bias-variance trade
• the Ridge closed form, and the exact shrinkage factor it applies
• the Lasso soft-thresholding rule, and why it produces genuine zeros
• both computed by hand on the orthogonal design from the Multiple Regression page
• the geometric reason a diamond selects features and a circle does not
• Elastic Net, and how to choose $\alpha$ honestly with cross-validation

Intuition

Ordinary least squares has exactly one goal: make the training residuals small. Given enough features it will happily grow enormous coefficients that cancel each other out — a coefficient of +4,000 on one column and −3,950 on a nearly identical one. The training fit looks superb, and the first new row destroys it.

Regularisation adds a second goal to the objective: keep the coefficients small. The model now has to justify every unit of coefficient with a real reduction in error. Noise cannot pay that price, so noise gets squeezed out. You trade a little bias for a large reduction in variance, and on held-out data the trade usually pays.

  flowchart LR
  A["Least squares cost
fit the data"] --> C["Penalised cost"]
  B["Penalty on coefficient size
stay simple"] --> C
  C --> R["Ridge (L2): shrink everything"]
  C --> L["Lasso (L1): shrink and select"]
  C --> E["Elastic Net: both"]

The math

The three objectives

Note the sums start at $j = 1$: the intercept is never penalised. Shrinking $\theta_0$ would tie the model’s predictions to zero rather than to the data’s mean, which is meaningless.

Ridge has a closed form

Differentiating $J_{\text{ridge}}$ and setting the gradient to zero gives a modified Normal Equation:

Adding $\lambda\mathbf{I}$ to the diagonal does something valuable beyond shrinkage: it makes the matrix invertible even when $\mathbf{X}^\top\mathbf{X}$ is singular. Ridge works when features outnumber samples, and when two columns are perfectly duplicated. Plain OLS does neither.

For an orthogonal design where $\mathbf{X}^\top\mathbf{X} = c\mathbf{I}$, the formula collapses to pure per-coefficient shrinkage:

Every coefficient is multiplied by the same factor below 1. That factor never reaches zero for finite $\lambda$ — which is the whole story of why Ridge cannot select features.

Lasso has no closed form, but it has a rule

The absolute value is not differentiable at zero, so there is no matrix formula. On an orthogonal design, though, the coordinate-wise solution is exact and beautifully simple — soft thresholding:

Subtract a fixed amount from every coefficient’s magnitude, and clamp at zero. Any coefficient smaller than the threshold becomes exactly zero, not merely small. That is feature selection falling out of the algebra.

Worked example by hand

The orthogonal design from Multiple Linear Regression, where $\mathbf{X}^\top\mathbf{X} = 4\mathbf{I}$, $\mathbf{X}^\top\mathbf{y} = [17, 9, 5]^\top$ and the OLS answer was $\boldsymbol{\theta} = [4.25,\ 2.25,\ 1.25]^\top$.

Ridge at $\lambda = 4$. With $c = 4$, the shrinkage factor is $4/(4+4) = 0.5$:

Ridge at $\lambda = 10$. Factor $4/14 \approx 0.286$:

Lasso at $\lambda = 4$. The threshold is $\lambda/(2c) = 4/8 = 0.5$, subtracted from each OLS coefficient:

Lasso at $\lambda = 10$. Threshold $10/8 = 1.25$:

There it is. At $\lambda = 10$ Lasso has deleted the weekend feature entirely, while Ridge at the same $\lambda$ merely shrank it to 0.357. The intercept stays at 4.25 throughout, because it is never penalised.

The geometry

figureWhy the shape of the penalty decides whether coefficients reach zeromatplotlib

Both penalties can be restated as a constrained problem: minimise the squared error subject to the coefficient vector staying inside a region of fixed size. The solution is where the growing elliptical cost contours first touch that region.

• The $\ell_1$ ball is a diamond, and its corners sit exactly on the axes. A corner is a point where one coordinate is zero. Corners stick out, so contours tend to hit them first.
• The $\ell_2$ ball is a circle with no corners anywhere. The touch point is generically a smooth spot where every coordinate is non-zero.

That is the entire explanation, and it generalises: in $n$ dimensions the $\ell_1$ ball has low-dimensional faces where many coefficients vanish at once.

See it move

Watch both penalties act on five coefficients as $\lambda$ sweeps upward. The Ridge bars shrink smoothly and never disappear; the Lasso bars hit zero one after another.

From scratch

Both rules, verified against scikit-learn on the four-row design:

import numpy as np
from sklearn.linear_model import Ridge, Lasso
 
X = np.array([[-1.0, -1.0], [-1.0, 1.0], [1.0, -1.0], [1.0, 1.0]])
y = np.array([1.0, 3.0, 5.0, 8.0])
c = 4.0                                  # X.T @ X = 4 * I for this design
ols = np.array([2.25, 1.25])             # from the Multiple Regression page
 
 
def ridge_closed_form(theta_ols, lam, c):
    return c / (c + lam) * theta_ols
 
 
def lasso_soft_threshold(theta_ols, lam, c):
    shrink = lam / (2 * c)
    return np.sign(theta_ols) * np.maximum(np.abs(theta_ols) - shrink, 0.0)
 
 
for lam in (4.0, 10.0):
    mine_r = ridge_closed_form(ols, lam, c)
    mine_l = lasso_soft_threshold(ols, lam, c)
 
    sk_r = Ridge(alpha=lam).fit(X, y).coef_
    sk_l = Lasso(alpha=lam / (2 * len(y)), max_iter=100000).fit(X, y).coef_
 
    print(f"lambda = {lam}")
    print(f"  ridge  mine {mine_r.round(4)}  sklearn {sk_r.round(4)}")
    print(f"  lasso  mine {mine_l.round(4)}  sklearn {sk_l.round(4)}")
 
# lambda = 4.0
#   ridge  mine [1.125 0.625]  sklearn [1.125 0.625]
#   lasso  mine [1.75 0.75  ]  sklearn [1.75 0.75  ]
# lambda = 10.0
#   ridge  mine [0.6429 0.3571]  sklearn [0.6429 0.3571]
#   lasso  mine [1. 0.]         sklearn [1. 0.]

import numpy as np
from sklearn.linear_model import Ridge, Lasso
 
X = np.array([[-1.0, -1.0], [-1.0, 1.0], [1.0, -1.0], [1.0, 1.0]])
y = np.array([1.0, 3.0, 5.0, 8.0])
c = 4.0                                  # X.T @ X = 4 * I for this design
ols = np.array([2.25, 1.25])             # from the Multiple Regression page
 
 
def ridge_closed_form(theta_ols, lam, c):
    return c / (c + lam) * theta_ols
 
 
def lasso_soft_threshold(theta_ols, lam, c):
    shrink = lam / (2 * c)
    return np.sign(theta_ols) * np.maximum(np.abs(theta_ols) - shrink, 0.0)
 
 
for lam in (4.0, 10.0):
    mine_r = ridge_closed_form(ols, lam, c)
    mine_l = lasso_soft_threshold(ols, lam, c)
 
    sk_r = Ridge(alpha=lam).fit(X, y).coef_
    sk_l = Lasso(alpha=lam / (2 * len(y)), max_iter=100000).fit(X, y).coef_
 
    print(f"lambda = {lam}")
    print(f"  ridge  mine {mine_r.round(4)}  sklearn {sk_r.round(4)}")
    print(f"  lasso  mine {mine_l.round(4)}  sklearn {sk_l.round(4)}")
 
# lambda = 4.0
#   ridge  mine [1.125 0.625]  sklearn [1.125 0.625]
#   lasso  mine [1.75 0.75  ]  sklearn [1.75 0.75  ]
# lambda = 10.0
#   ridge  mine [0.6429 0.3571]  sklearn [0.6429 0.3571]
#   lasso  mine [1. 0.]         sklearn [1. 0.]

Coefficient paths on real data

figureTen standardised diabetes features, penalised two waysmatplotlib

Reading the plots

1. Both start at the OLS solution. At the far left the penalty is negligible and both panels reproduce ordinary least squares.
2. Ridge shrinks in proportion. The largest coefficients fall fastest in absolute terms, but the ordering is broadly preserved and nothing ever reaches the axis.
3. Lasso eliminates in sequence. Each curve hits zero at its own $\alpha$ and stays there. The order in which features drop out is a crude but genuinely useful importance ranking.
4. Both end at zero. Push $\alpha$ high enough and every coefficient vanishes; the model becomes “always predict the mean”.

With scikit-learn

On the raw ten diabetes features, regularisation changes almost nothing — 442 samples over 10 well-behaved columns is simply not an overfitting situation. Expand to degree 2 (65 features) and the picture changes completely:

from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
 
X, y = load_diabetes(return_X_y=True)
 
 
def cv_mse(estimator):
    pipe = make_pipeline(
        PolynomialFeatures(2, include_bias=False),   # 10 features -> 65
        StandardScaler(),                            # penalties need a common scale
        estimator,
    )
    return -cross_val_score(
        pipe, X, y, cv=5, scoring="neg_mean_squared_error"
    ).mean()
 
 
print(f"OLS              {cv_mse(LinearRegression()):.1f}")   # OLS              3495.3
print(f"Ridge alpha=10   {cv_mse(Ridge(alpha=10)):.1f}")      # Ridge alpha=10   3165.1
print(f"Ridge alpha=100  {cv_mse(Ridge(alpha=100)):.1f}")     # Ridge alpha=100  3063.0
print(f"Lasso alpha=1    {cv_mse(Lasso(alpha=1, max_iter=200000)):.1f}")
#                                                             # Lasso alpha=1    3016.1

from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
 
X, y = load_diabetes(return_X_y=True)
 
 
def cv_mse(estimator):
    pipe = make_pipeline(
        PolynomialFeatures(2, include_bias=False),   # 10 features -> 65
        StandardScaler(),                            # penalties need a common scale
        estimator,
    )
    return -cross_val_score(
        pipe, X, y, cv=5, scoring="neg_mean_squared_error"
    ).mean()
 
 
print(f"OLS              {cv_mse(LinearRegression()):.1f}")   # OLS              3495.3
print(f"Ridge alpha=10   {cv_mse(Ridge(alpha=10)):.1f}")      # Ridge alpha=10   3165.1
print(f"Ridge alpha=100  {cv_mse(Ridge(alpha=100)):.1f}")     # Ridge alpha=100  3063.0
print(f"Lasso alpha=1    {cv_mse(Lasso(alpha=1, max_iter=200000)):.1f}")
#                                                             # Lasso alpha=1    3016.1

Regularisation cuts cross-validated error by roughly 14%, from 3495 to 3016 — and the Lasso model gets there using 34 of the 65 columns.

figureChoosing alpha by cross-validationmatplotlib

Choosing $\alpha$ properly

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import RidgeCV, LassoCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
 
X, y = load_diabetes(return_X_y=True)
alphas = np.geomspace(1e-3, 1e3, 60)
 
ridge = make_pipeline(StandardScaler(), RidgeCV(alphas=alphas)).fit(X, y)
lasso = make_pipeline(
    StandardScaler(), LassoCV(alphas=alphas, cv=5, max_iter=100000, random_state=0)
).fit(X, y)
 
print(f"RidgeCV alpha = {ridge[-1].alpha_:.4f}")           # RidgeCV alpha = 1.7957
print(f"LassoCV alpha = {lasso[-1].alpha_:.4f}")           # LassoCV alpha = 0.0732
print(f"Lasso kept {(lasso[-1].coef_ != 0).sum()} of 10")  # Lasso kept 9 of 10

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import RidgeCV, LassoCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
 
X, y = load_diabetes(return_X_y=True)
alphas = np.geomspace(1e-3, 1e3, 60)
 
ridge = make_pipeline(StandardScaler(), RidgeCV(alphas=alphas)).fit(X, y)
lasso = make_pipeline(
    StandardScaler(), LassoCV(alphas=alphas, cv=5, max_iter=100000, random_state=0)
).fit(X, y)
 
print(f"RidgeCV alpha = {ridge[-1].alpha_:.4f}")           # RidgeCV alpha = 1.7957
print(f"LassoCV alpha = {lasso[-1].alpha_:.4f}")           # LassoCV alpha = 0.0732
print(f"Lasso kept {(lasso[-1].coef_ != 0).sum()} of 10")  # Lasso kept 9 of 10

RidgeCVRidgeCV and LassoCVLassoCV do the sweep internally and are considerably faster than a manual GridSearchCVGridSearchCV because they reuse the solution path.

Which penalty

Elastic Net exists because Lasso has a specific failure mode: with two nearly identical features it picks one at random and zeroes the other, and which one it picks flips with a different random seed. Adding a little L2 to the L1 makes correlated features enter or leave together.

Pitfalls

🧪 Try It Yourself

Exercise 1 – The Ridge shrinkage factor

Exercise 2 – Soft thresholding by hand

Exercise 3 – Ridge keeps everything, Lasso does not

Exercise 4 – Regularisation earns its keep on wide data

Exercise 5 – Let cross-validation pick alpha

Recap

• Regularisation adds a penalty on coefficient size, trading a little bias for a lot less variance.
• Ridge has the closed form $(\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{y}$, which also fixes singular designs.
• On an orthogonal design Ridge multiplies each coefficient by $c/(c+\lambda)$; Lasso subtracts $\lambda/2c$ from each magnitude and clamps at zero.
• Worked by hand at $\lambda = 10$: Ridge gives $[0.643, 0.357]$, Lasso gives $[1.0, 0]$ — one feature deleted.
• The diamond has corners on the axes and the circle does not. That is the whole geometric story.
• On 65 polynomial features, cross-validated MSE drops from 3495 (OLS) to 3063 (Ridge) to 3016 (Lasso, using 34 columns). On the raw 10 features it changes nothing.
• Always standardise, never penalise the intercept, and always cross-validate $\alpha$.

Exercise 6 – Watch each penalty act on the same coefficients

Next

Continue to Metrics - R-Squared and Adjusted R-Squared — how to report all of this to someone who was not in the room.