Bias vs Variance Tradeoff

“Underfitting” and “overfitting” are the symptoms; bias and variance are the disease. Every prediction error a model makes can be split into three ingredients, and two of them are under your control. Understanding which one is hurting you tells you exactly what to fix — more data, a simpler model, a more flexible one, or nothing at all (some error is just noise).

Two ways models can be wrong

Bias (systematic error)

High bias means the model is too simple and misses real patterns.

underfits
training error is high
validation error is high

Variance (sensitivity to noise)

High variance means the model is too complex and learns noise.

overfits
training error is low
validation error is high

diagram Diagram mermaid

  flowchart LR
  B[High bias] --> U[Underfitting]
  V[High variance] --> O[Overfitting]

The tradeoff

As model complexity increases:

bias tends to decrease
variance tends to increase

Goal: find a sweet spot.

How to reduce bias

add more features
use a more flexible model
reduce regularization

How to reduce variance

collect more data
increase regularization
simplify the model
use bagging/ensembles

Runnable example: watching bias and variance move

Train the same polynomial model on several different random samples of noisy data. Bias shows up as how far the average prediction is from the truth; variance shows up as how much the predictions disagree with each other across samples:

bias_variance_demo.py

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
 
rng = np.random.RandomState(0)
x_test = np.array([[0.0]])          # the single point we probe
true_value = 2.0                     # f(0) = 2 for y = 0.5x^2 + x + 2
 
def sample_predictions(degree, n_datasets=200, n_points=30):
    preds = []
    for _ in range(n_datasets):
        X = 6 * rng.rand(n_points, 1) - 3
        y = 0.5 * X**2 + X + 2 + rng.randn(n_points, 1) * 0.5
        model = Pipeline([
            ("poly", PolynomialFeatures(degree=degree, include_bias=False)),
            ("lin_reg", LinearRegression()),
        ])
        model.fit(X, y)
        preds.append(model.predict(x_test)[0, 0])
    preds = np.array(preds)
    bias_sq = (preds.mean() - true_value) ** 2
    variance = preds.var()
    return bias_sq, variance
 
for degree in [1, 2, 15]:
    bias_sq, variance = sample_predictions(degree)
    print(f"degree={degree:>2}  bias^2={bias_sq:.3f}  variance={variance:.3f}")

bias_variance_demo.py

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
 
rng = np.random.RandomState(0)
x_test = np.array([[0.0]])          # the single point we probe
true_value = 2.0                     # f(0) = 2 for y = 0.5x^2 + x + 2
 
def sample_predictions(degree, n_datasets=200, n_points=30):
    preds = []
    for _ in range(n_datasets):
        X = 6 * rng.rand(n_points, 1) - 3
        y = 0.5 * X**2 + X + 2 + rng.randn(n_points, 1) * 0.5
        model = Pipeline([
            ("poly", PolynomialFeatures(degree=degree, include_bias=False)),
            ("lin_reg", LinearRegression()),
        ])
        model.fit(X, y)
        preds.append(model.predict(x_test)[0, 0])
    preds = np.array(preds)
    bias_sq = (preds.mean() - true_value) ** 2
    variance = preds.var()
    return bias_sq, variance
 
for degree in [1, 2, 15]:
    bias_sq, variance = sample_predictions(degree)
    print(f"degree={degree:>2}  bias^2={bias_sq:.3f}  variance={variance:.3f}")

text

degree= 1  bias^2=0.482  variance=0.019
degree= 2  bias^2=0.001  variance=0.031
degree=15  bias^2=0.006  variance=1.744

text

degree= 1  bias^2=0.482  variance=0.019
degree= 2  bias^2=0.001  variance=0.031
degree=15  bias^2=0.006  variance=1.744

Notice the shape: the straight line (degree=1degree=1) has the highest bias but low variance — a stable, wrong answer. The degree-15 polynomial has near-zero bias but its predictions swing wildly from dataset to dataset — high variance. Degree 2 (the true shape of the data) minimizes both.

Visualize it

The classic dartboard view: each target represents where predictions land across many different training sets, with the bullseye being the true value. Low bias means shots cluster near the bullseye; low variance means shots cluster tightly together — these are independent axes:

sketch Bias vs variance dartboard p5.js

Darts are thrown one at a time onto each board, then the round fades and rethrows — bias controls how close to the bullseye, variance controls how tight the cluster is.

Mini-checkpoint

If training and validation accuracy are both low:

bias or variance?

(Usually bias / underfitting.)

🧪 Try It Yourself

Exercise 1 – Measure Bias with Average Prediction

Exercise 2 – Measure Variance Across Models

Exercise 3 – Choose the Fix

Continue to K-Fold Cross-Validation — a more reliable way to actually measure bias and variance than a single train/validation split.

If this helped you, consider buying me a coffee ☕

Buy me a coffee