Every model you train will be wrong in one of two directions: too simple to see the pattern, or so flexible it memorizes the noise. Learning to tell these apart from a handful of numbers — train score, validation score — is the single most useful diagnostic skill in machine learning, and it’s the topic that opens every serious tuning workflow.

Underfitting

Underfitting happens when the model is too simple.

Symptoms:

• training score is poor
• validation score is also poor

Fixes:

• add features
• use a stronger model (e.g., trees/boosting)
• lower regularization

Overfitting

Overfitting happens when the model learns noise.

Symptoms:

• training score is great
• validation/test score is much worse

Fixes:

• simplify model
• add regularization
• get more data
• use cross-validation

Visual intuition

  flowchart TD
  A[Model complexity] --> B["Underfit (too simple)"]
  A --> C[Good fit]
  A --> D["Overfit (too complex)"]

Runnable example: comparing model complexity

The clearest way to see under/overfitting is to fit the same data with models of increasing complexity (via PolynomialFeaturesPolynomialFeatures) and compare train vs. validation error at each degree:

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
 
rng = np.random.RandomState(42)
X = 6 * rng.rand(100, 1) - 3
y = 0.5 * X**2 + X + 2 + rng.randn(100, 1)
 
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
 
for degree in [1, 2, 300]:
    model = Pipeline([
        ("poly", PolynomialFeatures(degree=degree, include_bias=False)),
        ("lin_reg", LinearRegression()),
    ])
    model.fit(X_train, y_train)
 
    train_rmse = mean_squared_error(y_train, model.predict(X_train)) ** 0.5
    val_rmse = mean_squared_error(y_val, model.predict(X_val)) ** 0.5
    gap = val_rmse - train_rmse
 
    label = "underfit" if train_rmse > 1.0 else ("overfit" if gap > 0.5 else "good fit")
    print(f"degree={degree:>3}  train_rmse={train_rmse:.3f}  val_rmse={val_rmse:.3f}  -> {label}")

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
 
rng = np.random.RandomState(42)
X = 6 * rng.rand(100, 1) - 3
y = 0.5 * X**2 + X + 2 + rng.randn(100, 1)
 
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
 
for degree in [1, 2, 300]:
    model = Pipeline([
        ("poly", PolynomialFeatures(degree=degree, include_bias=False)),
        ("lin_reg", LinearRegression()),
    ])
    model.fit(X_train, y_train)
 
    train_rmse = mean_squared_error(y_train, model.predict(X_train)) ** 0.5
    val_rmse = mean_squared_error(y_val, model.predict(X_val)) ** 0.5
    gap = val_rmse - train_rmse
 
    label = "underfit" if train_rmse > 1.0 else ("overfit" if gap > 0.5 else "good fit")
    print(f"degree={degree:>3}  train_rmse={train_rmse:.3f}  val_rmse={val_rmse:.3f}  -> {label}")

degree=  1  train_rmse=1.084  val_rmse=1.121  -> underfit
degree=  2  train_rmse=0.789  val_rmse=0.798  -> good fit
degree=300  train_rmse=0.601  val_rmse=1.940  -> overfit

degree=  1  train_rmse=1.084  val_rmse=1.121  -> underfit
degree=  2  train_rmse=0.789  val_rmse=0.798  -> good fit
degree=300  train_rmse=0.601  val_rmse=1.940  -> overfit

Visualize it

The same noisy data, three models. Underfitting is too simple to capture the pattern; overfitting is so flexible it chases the random noise; the good fit captures the real trend and ignores the wobble — the balance you’re always aiming for. Watch the fitted curve itself morph between all three as “model complexity” slides from simple to complex and back:

Learning curves: the diagnostic tool

The book’s other trick — used before cross-validation scores are even available — is to plot learning curves: training and validation error as a function of how many training examples the model has seen so far. Train the model on growing subsets of the training set (1 example, 2 examples, 3, …) and record both errors at each size.

• Underfitting learning curves: both curves rise to a plateau and end up close together — but the plateau is high. More data won’t help; you need a better model.
• Overfitting learning curves: training error stays low, but there’s a persistent gap to a much higher validation error. More data (or regularization) narrows the gap.

Mini-checkpoint

Your model has:

• train accuracy: 99%
• validation accuracy: 75%

What is happening?

(Overfitting.)

🧪 Try It Yourself

Exercise 1 – Build a Polynomial Pipeline

Exercise 2 – Spot the Overfit Gap

Exercise 3 – Fix an Underfitting Model

Next

Continue to Bias vs Variance Tradeoff — the formal, quantitative version of the same under/overfit story.