What R² measures

Every model in this phase — simple, multiple, polynomial, Ridge, Lasso — is trained by minimizing a cost function like MSE. But MSE is in squared units (squared dollars, squared degrees), which is awkward to reason about on its own. R² (the coefficient of determination) answers a more intuitive question instead:

“Compared to just guessing the mean every time, how much of the variance in the target did my model actually explain?”

• R² = 1 → perfect predictions, every point lands exactly on the model
• R² = 0 → the model is no better than always predicting the mean of yy
• R² < 0 → the model is worse than that trivial baseline (yes, this can happen on a bad fit or a badly generalizing test set)

The intuition: two sums of squares

R² compares two sources of error:

• SS_res (residual sum of squares) — how far your model’s predictions are from the actual values: Σ(yi - ŷi)²Σ(yi - ŷi)². This is the numerator of MSE, unaveraged.
• SS_tot (total sum of squares) — how far the actual values are from their own mean: Σ(yi - ȳ)²Σ(yi - ȳ)². This is the error of the “always predict the mean” baseline you met back in Introduction to Regression Analysis.

  flowchart TD
  Y["True values y"] --> M["Mean baseline (ȳ)"]
  Y --> F["Fitted model (ŷ)"]
  M --> SST["SS_tot = Σ(y - ȳ)²
(total variance)"]
  F --> SSR["SS_res = Σ(y - ŷ)²
(unexplained variance)"]
  SST --> R2["R² = 1 - SS_res / SS_tot"]
  SSR --> R2

The formula:

R² = 1 - (SS_res / SS_tot)R² = 1 - (SS_res / SS_tot)

If your model’s residual error (SS_resSS_res) is tiny compared to the total variance (SS_totSS_tot), R² approaches 1 — you’ve explained almost all the spread in the data. If your model does no better than the mean, SS_res ≈ SS_totSS_res ≈ SS_tot and R² collapses to 0.

Scikit-learn example

from sklearn.metrics import r2_score
 
y_true = [3, 5, 2, 7, 8]
y_pred = [2.8, 5.1, 1.9, 7.3, 7.6]
 
r2 = r2_score(y_true, y_pred)
print("R2:", round(r2, 4))

from sklearn.metrics import r2_score
 
y_true = [3, 5, 2, 7, 8]
y_pred = [2.8, 5.1, 1.9, 7.3, 7.6]
 
r2 = r2_score(y_true, y_pred)
print("R2:", round(r2, 4))

And here’s what scikit-learn is doing under the hood — the same two sums of squares from the diagram above, computed by hand with NumPy:

import numpy as np
 
y_true = np.array([3, 5, 2, 7, 8])
y_pred = np.array([2.8, 5.1, 1.9, 7.3, 7.6])
 
ss_res = np.sum((y_true - y_pred) ** 2)          # model's leftover error
ss_tot = np.sum((y_true - y_true.mean()) ** 2)   # mean-baseline error
 
r2 = 1 - ss_res / ss_tot
print("SS_res:", round(ss_res, 3))
print("SS_tot:", round(ss_tot, 3))
print("R2:", round(r2, 4))

import numpy as np
 
y_true = np.array([3, 5, 2, 7, 8])
y_pred = np.array([2.8, 5.1, 1.9, 7.3, 7.6])
 
ss_res = np.sum((y_true - y_pred) ** 2)          # model's leftover error
ss_tot = np.sum((y_true - y_true.mean()) ** 2)   # mean-baseline error
 
r2 = 1 - ss_res / ss_tot
print("SS_res:", round(ss_res, 3))
print("SS_tot:", round(ss_tot, 3))
print("R2:", round(r2, 4))

Why R² can be misleading

• Adding more features almost never decreases R² — even a completely useless, random feature can nudge it up slightly, simply because the model has one more knob to fit noise with. This is the same combinatorial-feature trap you saw with PolynomialFeaturesPolynomialFeatures: more parameters, more chances to overfit.
• R² doesn’t tell you the error’s real-world size. A model can post R² = 0.95 and still be off by thousands of dollars on individual predictions if the target’s overall variance is huge. Always pair R² with RMSE or MAE, which stay in the original units.
• R² on the training set can look great purely from overfitting — exactly the pattern the learning curves in the Polynomial Regression lesson exposed: a high-degree model fits training points almost perfectly (R² near 1 on training data) while its validation R² tells a very different story.

Adjusted R²

Because plain R² rewards adding features regardless of whether they help, adjusted R² applies a penalty based on the number of predictors kk relative to the sample size nn:

Adjusted R² = 1 - [(1 - R²) · (n - 1) / (n - k - 1)]Adjusted R² = 1 - [(1 - R²) · (n - 1) / (n - k - 1)]

• Adding a genuinely useful feature raises adjusted R².
• Adding a useless feature lowers adjusted R² (the penalty outweighs the tiny R² gain).
• It’s the right tool for comparing models with different numbers of predictors — for example, deciding whether a second-degree or a fifth-degree polynomial expansion is actually worth the extra terms.

import numpy as np
from sklearn.metrics import r2_score
 
y_true = np.array([3, 5, 2, 7, 8, 6, 4])
y_pred = np.array([2.8, 5.1, 1.9, 7.3, 7.6, 6.2, 3.9])
 
n = len(y_true)   # number of samples
k = 2             # number of predictors (features) used by the model
 
r2 = r2_score(y_true, y_pred)
adj_r2 = 1 - (1 - r2) * (n - 1) / (n - k - 1)
 
print("R2:", round(r2, 4))
print("Adjusted R2:", round(adj_r2, 4))

import numpy as np
from sklearn.metrics import r2_score
 
y_true = np.array([3, 5, 2, 7, 8, 6, 4])
y_pred = np.array([2.8, 5.1, 1.9, 7.3, 7.6, 6.2, 3.9])
 
n = len(y_true)   # number of samples
k = 2             # number of predictors (features) used by the model
 
r2 = r2_score(y_true, y_pred)
adj_r2 = 1 - (1 - r2) * (n - 1) / (n - k - 1)
 
print("R2:", round(r2, 4))
print("Adjusted R2:", round(adj_r2, 4))

Visualize it

Watch the residual (leftover) error shrink as the fitted line improves from a flat mean-only baseline to a good fit: the tall red segments are each point’s distance to the mean (SS_totSS_tot), and the short green segments are its distance to the current model (SS_resSS_res). R² is just 1 -1 - the ratio of the green area to the red area.

What metrics to use in practice

Use a combination, not just one:

• MAE (mean absolute error) — easiest to explain to stakeholders, same units as yy
• RMSE — punishes large errors more, same units as yy
• R² / adjusted R² — how much of the variance is explained, unitless and easy to compare across targets, but not sufficient alone

Mini-checkpoint

If your model has R² = 0.92 but RMSE is huge, what does that imply?

• It explains variance well relative to a flat baseline, but individual predictions can still be too inaccurate for real use — always check RMSE/MAE alongside R².

🧪 Try It Yourself

Exercise 1 – Compute R² with Scikit-Learn

Exercise 2 – R² from the Two Sums of Squares

Exercise 3 – Adjust R² for the Number of Features

Next

This wraps up Phase 3. Continue to Phase 4 - Supervised Learning - Classification to learn Logistic Regression’s other half — predicting categories instead of numbers — plus KNN, SVMs, Decision Trees, and Naive Bayes.