Skip to content
Python Central Hub

Polynomial Regression

Why polynomial regression exists

Sometimes the relationship is curved:

  • learning curves
  • growth/decay
  • diminishing returns

A straight line underfits.

Polynomial regression keeps a linear model but expands features:

  • x → [x, x², x³, …]

Picture intuition

diagram Diagram mermaid

Scikit-learn example

Polynomial regression
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
 
X = np.array([0, 1, 2, 3, 4]).reshape(-1, 1)
y = np.array([1, 2, 5, 10, 17])  # roughly quadratic
 
poly_model = Pipeline(
    steps=[
        ("poly", PolynomialFeatures(degree=2, include_bias=False)),
        ("lin", LinearRegression()),
    ]
)
 
poly_model.fit(X, y)
print(poly_model.predict([[5]]))
Polynomial regression
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
 
X = np.array([0, 1, 2, 3, 4]).reshape(-1, 1)
y = np.array([1, 2, 5, 10, 17])  # roughly quadratic
 
poly_model = Pipeline(
    steps=[
        ("poly", PolynomialFeatures(degree=2, include_bias=False)),
        ("lin", LinearRegression()),
    ]
)
 
poly_model.fit(X, y)
print(poly_model.predict([[5]]))

This is exactly the recipe from Hands-On Machine Learning: generate some quadratic-looking data, transform it with PolynomialFeaturesPolynomialFeatures, then hand the expanded features to an ordinary LinearRegressionLinearRegression:

PolynomialFeatures on quadratic data
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
 
np.random.seed(42)
m = 100
X = 6 * np.random.rand(m, 1) - 3
y = 0.5 * X**2 + X + 2 + np.random.randn(m, 1)
 
poly_features = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly_features.fit_transform(X)
 
print("original feature:", X[0])
print("expanded features:", X_poly[0])   # [x, x^2]
 
lin_reg = LinearRegression()
lin_reg.fit(X_poly, y)
print("intercept:", lin_reg.intercept_, "coef:", lin_reg.coef_)
# ≈ y = 0.56x^2 + 0.93x + 1.78, close to the true 0.5x^2 + x + 2
PolynomialFeatures on quadratic data
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
 
np.random.seed(42)
m = 100
X = 6 * np.random.rand(m, 1) - 3
y = 0.5 * X**2 + X + 2 + np.random.randn(m, 1)
 
poly_features = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly_features.fit_transform(X)
 
print("original feature:", X[0])
print("expanded features:", X_poly[0])   # [x, x^2]
 
lin_reg = LinearRegression()
lin_reg.fit(X_poly, y)
print("intercept:", lin_reg.intercept_, "coef:", lin_reg.coef_)
# ≈ y = 0.56x^2 + 0.93x + 1.78, close to the true 0.5x^2 + x + 2

PolynomialFeatures(degree=d)PolynomialFeatures(degree=d) doesn’t just add powers of one feature — with multiple input features it also adds every combination of features up to that degree (e.g. a²a², abab, a²ba²b, b³b³, …). Careful: the feature count grows combinatorially, following (n + d)! / (d! n!)(n + d)! / (d! n!) for nn original features. Doubling the degree can explode the feature count.

Overfitting warning

Higher degrees can fit noise. Figure 4-14 in the book compares a 300-degree polynomial (wiggles to touch almost every point — overfitting), a plain line (underfitting), and a quadratic model (fits the true shape of the data, since it was generated by a quadratic function).

Use:

  • validation
  • regularization (Ridge/Lasso)

Learning curves: telling over/underfitting apart

You usually don’t know the “true” degree that generated your data. The book’s tool for diagnosing over/underfitting is the learning curve: plot the model’s training error and validation error as the training set size grows.

Plotting learning curves (book recipe)
import numpy as np
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
 
def plot_learning_curves(model, X, y):
    X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
    train_errors, val_errors = [], []
    for m in range(1, len(X_train)):
        model.fit(X_train[:m], y_train[:m])
        y_train_predict = model.predict(X_train[:m])
        y_val_predict = model.predict(X_val)
        train_errors.append(mean_squared_error(y_train[:m], y_train_predict))
        val_errors.append(mean_squared_error(y_val, y_val_predict))
    return np.sqrt(train_errors), np.sqrt(val_errors)
 
# lin_reg = LinearRegression(); plot_learning_curves(lin_reg, X, y)
# → both curves plateau close together and high = underfitting
#
# polynomial_regression = Pipeline([
#     ("poly_features", PolynomialFeatures(degree=10, include_bias=False)),
#     ("lin_reg", LinearRegression()),
# ])
# plot_learning_curves(polynomial_regression, X, y)
# → training error much lower, but a persistent gap to validation error = overfitting
Plotting learning curves (book recipe)
import numpy as np
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
 
def plot_learning_curves(model, X, y):
    X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
    train_errors, val_errors = [], []
    for m in range(1, len(X_train)):
        model.fit(X_train[:m], y_train[:m])
        y_train_predict = model.predict(X_train[:m])
        y_val_predict = model.predict(X_val)
        train_errors.append(mean_squared_error(y_train[:m], y_train_predict))
        val_errors.append(mean_squared_error(y_val, y_val_predict))
    return np.sqrt(train_errors), np.sqrt(val_errors)
 
# lin_reg = LinearRegression(); plot_learning_curves(lin_reg, X, y)
# → both curves plateau close together and high = underfitting
#
# polynomial_regression = Pipeline([
#     ("poly_features", PolynomialFeatures(degree=10, include_bias=False)),
#     ("lin_reg", LinearRegression()),
# ])
# plot_learning_curves(polynomial_regression, X, y)
# → training error much lower, but a persistent gap to validation error = overfitting
  • Underfitting: both curves plateau at a high error, close together. Adding more training data won’t help — you need a more complex model or better features.
  • Overfitting: training error is low, but there’s a gap to a higher validation error. Feeding the model more training data narrows this gap.

This connects directly to the bias/variance trade-off: a high-bias model (too simple) underfits; a high-variance model (too complex, like a high-degree polynomial) overfits. Increasing model complexity trades bias for variance.

Visualize it

The chart on the left animates three model complexities fitting the same noisy quadratic data — watch the line underfit, fit well, and then overfit (wiggle wildly through every point). The chart on the right shows the matching learning curves: notice how the training/validation gap opens up exactly when the model starts overfitting.

sketch Under/over-fitting a curved dataset p5.js
A degree-1 line underfits, a degree-2 curve fits well, and a high-degree polynomial overfits by wiggling through the noise.

Mini-checkpoint

Try degrees 1, 2, 3.

  • does validation error improve?
  • does it start getting worse at high degree?

🧪 Try It Yourself

Exercise 1 – Expand Features with PolynomialFeatures

Exercise 2 – Build a Polynomial Pipeline

Exercise 3 – Spot Overfitting with Train vs Validation MSE

Next

Continue to Cost Functions - Mean Squared Error (MSE) — formalize the error measure that every model in this phase is trying to minimize.

If this helped you, consider buying me a coffee ☕

Buy me a coffee

Was this page helpful?

Let us know how we did