What you’ll learn

• the three scaler formulas, each computed by hand on the same five numbers
• which models are destroyed by unscaled features, which are indifferent, and why — measured
• what a single outlier does to each scaler, quantified
• why fitfit belongs to the training set and transformtransform to everything
• when to log-transform instead of scale
• that scaling never changes the shape of a distribution

Intuition

The housing data has median_incomemedian_income spanning 0.5 to 15 and populationpopulation spanning 3 to 35,682. Nothing is wrong with either column. The problem is that some algorithms treat “a difference of 1” as meaning the same thing in both.

Two model families care intensely:

• Distance-based — KNN, SVM with RBF, k-means. Euclidean distance sums squared differences, so a feature with a range in the thousands drowns one with a range of 10.
• Gradient-based — linear and logistic regression via SGD, neural networks. Unequal feature scales stretch the cost surface into a ravine, and descent zig-zags instead of descending.

Two families do not:

• Tree-based — splits depend on the ordering of values, and scaling is monotone.
• Closed-form solvers — the Normal Equation solves exactly regardless of conditioning, within numerical limits.

  flowchart TD
  A["Feature ranges differ
by orders of magnitude"] --> B{"How does the model
use the numbers?"}
  B -->|"distances"| C["KNN, SVM, k-means
SCALING REQUIRED"]
  B -->|"iterative gradients"| D["SGD, neural networks
SCALING REQUIRED"]
  B -->|"value ordering"| E["Trees, forests, boosting
no effect"]
  B -->|"closed form"| F["Normal Equation
no effect"]

The math

StandardScalerStandardScaler uses the population standard deviation ($\div n$, not $\div (n-1)$), which is why hand calculations must use $\sigma$ rather than pandas’ default $s$.

Worked example by hand

Five values: $[1, 2, 3, 4, 5]$.

Step 1 — the statistics.

Step 2 — apply each formula.

Step 3 — check the properties. Min-max lands exactly on $[0, 1]$. Standardised values have mean 0 and population standard deviation 1. Robust values are centred on the median with the interquartile range as the unit.

Step 4 — note what did not change. All three columns are perfectly evenly spaced, exactly as the original was. Scaling is an affine map: it slides and stretches the axis and can do nothing else.

figureThe same column, four treatmentsmatplotlib

What one outlier does

figureValues 1 to 10, plus one reading of 1000matplotlib

Reading the plot

1. Min-max is catastrophic. The single value of 1000 defines the range, so the ten real observations occupy 0.9% of the output interval. Any distance computed between them is now effectively zero.
2. Standardisation is bad but less so. The outlier inflates $\sigma$, compressing everything else into a span of 0.03.
3. RobustScaler keeps the real data readable. Median and IQR ignore the extreme value entirely, so the ten normal points keep a usable spread — and the outlier is left at 198.8, visibly flagged rather than silently absorbed.

RobustScaler is the right default whenever outliers are present and cannot be removed.

See it move

Two features, wildly different ranges. Watch the cloud transform as the scalers are applied — and watch the shape stay exactly where it was.

Which models actually care

figureFive models, with and without scalingmatplotlib

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression, SGDRegressor
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsRegressor
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVR
 
housing = pd.read_csv(URL).dropna()
X = housing.select_dtypes("number").drop(columns=["median_house_value"]).values
y = housing["median_house_value"].values
sample = np.random.default_rng(0).choice(len(y), 3000, replace=False)
X, y = X[sample], y[sample]
 
models = {
    "linear": LinearRegression(),
    "sgd": SGDRegressor(max_iter=2000, tol=1e-4, random_state=0),
    "knn": KNeighborsRegressor(10),
    "svr": SVR(),
    "forest": RandomForestRegressor(n_estimators=60, random_state=0, n_jobs=-1),
}
 
for name, model in models.items():
    raw = -cross_val_score(model, X, y, cv=3,
                           scoring="neg_root_mean_squared_error").mean()
    scaled = -cross_val_score(make_pipeline(StandardScaler(), model), X, y, cv=3,
                              scoring="neg_root_mean_squared_error").mean()
    print(f"{name:<7} raw {raw:>22,.0f}   scaled {scaled:>9,.0f}   "
          f"{100 * (scaled - raw) / raw:+7.1f}%")
 
# linear  raw                 69,721   scaled    69,721      +0.0%
# sgd     raw  3,795,293,450,055,030   scaled    70,035    -100.0%
# knn     raw                100,607   scaled    69,128     -31.3%
# svr     raw                117,412   scaled   117,374      -0.0%
# forest  raw                 58,226   scaled    58,263      +0.1%

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression, SGDRegressor
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsRegressor
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVR
 
housing = pd.read_csv(URL).dropna()
X = housing.select_dtypes("number").drop(columns=["median_house_value"]).values
y = housing["median_house_value"].values
sample = np.random.default_rng(0).choice(len(y), 3000, replace=False)
X, y = X[sample], y[sample]
 
models = {
    "linear": LinearRegression(),
    "sgd": SGDRegressor(max_iter=2000, tol=1e-4, random_state=0),
    "knn": KNeighborsRegressor(10),
    "svr": SVR(),
    "forest": RandomForestRegressor(n_estimators=60, random_state=0, n_jobs=-1),
}
 
for name, model in models.items():
    raw = -cross_val_score(model, X, y, cv=3,
                           scoring="neg_root_mean_squared_error").mean()
    scaled = -cross_val_score(make_pipeline(StandardScaler(), model), X, y, cv=3,
                              scoring="neg_root_mean_squared_error").mean()
    print(f"{name:<7} raw {raw:>22,.0f}   scaled {scaled:>9,.0f}   "
          f"{100 * (scaled - raw) / raw:+7.1f}%")
 
# linear  raw                 69,721   scaled    69,721      +0.0%
# sgd     raw  3,795,293,450,055,030   scaled    70,035    -100.0%
# knn     raw                100,607   scaled    69,128     -31.3%
# svr     raw                117,412   scaled   117,374      -0.0%
# forest  raw                 58,226   scaled    58,263      +0.1%

Reading the table

• SGD is the headline. Raw features produce an RMSE of $3.8 \times 10^{15}$ — the optimiser diverged. Scaling turns a broken model into a working one, matching the closed-form solution to within 0.5%. Not an improvement; a repair.
• KNN gains 31%. Distance was being dominated by populationpopulation, which spans 35,000, while median_incomemedian_income spanning 15 contributed almost nothing.
• Linear regression is byte-identical. LinearRegressionLinearRegression uses lstsqlstsq, and least squares is invariant to affine feature transformations.
• The forest is unchanged to within noise, exactly as the theory says.
• SVR barely moves here only because this sample happens to be dominated by one large-range feature either way; on the full dataset with an RBF kernel, scaling matters a great deal.

Fit on train, transform everything

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
 
X_train, X_test = train_test_split(X, test_size=0.2, random_state=42)
 
scaler = StandardScaler()
scaler.fit(X_train)                    # learn mu and sigma from training data only
 
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)     # NOT fit_transform
 
print(X_train_scaled.mean(axis=0).round(6))  # ~0 by construction
print(X_test_scaled.mean(axis=0).round(4))   # near 0, but not exactly — correct

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
 
X_train, X_test = train_test_split(X, test_size=0.2, random_state=42)
 
scaler = StandardScaler()
scaler.fit(X_train)                    # learn mu and sigma from training data only
 
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)     # NOT fit_transform
 
print(X_train_scaled.mean(axis=0).round(6))  # ~0 by construction
print(X_test_scaled.mean(axis=0).round(4))   # near 0, but not exactly — correct

The test set’s scaled mean is not exactly zero, and that is the point. It was scaled by parameters learned elsewhere, exactly as a live request will be. If your test data scales to a perfect mean of zero, you called fit_transformfit_transform on it and leaked.

Scaling is not transforming

Scaling cannot fix skew, because it is affine. Six of the nine housing columns are heavily right-skewed, and no scaler touches that. A logarithm does:

import numpy as np
from scipy import stats
 
col = housing["population"]
print("raw skew:", round(float(stats.skew(col)), 3))            # 4.935
print("scaled skew:", round(float(stats.skew(
    (col - col.mean()) / col.std())), 3))                       # 4.935 — unchanged
print("log skew:", round(float(stats.skew(np.log1p(col))), 3))  # -1.044

import numpy as np
from scipy import stats
 
col = housing["population"]
print("raw skew:", round(float(stats.skew(col)), 3))            # 4.935
print("scaled skew:", round(float(stats.skew(
    (col - col.mean()) / col.std())), 3))                       # 4.935 — unchanged
print("log skew:", round(float(stats.skew(np.log1p(col))), 3))  # -1.044

Standardising leaves the skew at 4.94. log1plog1p brings it to −1.04 — overshooting into a mild left skew, which is a reminder that a log is a fixed transform, not a tuned one. Different tools for different problems: scalers equalise ranges, transforms reshape distributions.

Pitfalls

🧪 Try It Yourself

Exercise 1 – Min-max scale by hand

Exercise 2 – Standardise, and match scikit-learn

Exercise 3 – Watch one outlier crush the rest

Exercise 4 – Fit on train, transform both

Exercise 5 – Scaling cannot fix skew

Recap

• Min-max maps to $[0,1]$, standardisation to mean 0 and sd 1, robust to median 0 and IQR 1.
• Worked by hand on $[1,2,3,4,5]$: all three produce evenly spaced output, because scaling is affine.
• Distance-based and gradient-based models need scaling; trees and closed-form solvers do not. SGD on raw housing features reaches an RMSE of $3.8\times10^{15}$; scaled, it matches the exact solution.
• One outlier crushes the ten real values into a span of 0.009 under min-max and 0.032 under standardisation, against 1.8 under RobustScalerRobustScaler.
• Fit on train, transform on everything. A test set that scales to exactly zero mean is evidence of a leak.
• Scaling cannot change skewness — 4.935 before and after. Use log1plog1p or PowerTransformerPowerTransformer for that.

Exercise 6 – Which scaler survives one outlier?

Next

Continue to Transformation Pipelines & Custom Transformers — assemble imputing, encoding and scaling into one object that cannot leak.