Phase 3 - Supervised Learning - Regression

Regression is where supervised learning becomes concrete. Everything in this phase — the cost function, the gradient, the penalty term, the held-out score — reappears unchanged in classification, in ensembles, and in every neural network in the Deep Learning module. Learn it properly once here and the rest of the curriculum is mostly variations.

What this phase covers

Eight pages, in dependency order. Each builds on the one before, and each hand-works its arithmetic on the same five-point dataset so the numbers stay comparable across the whole phase.

#	Page	Core idea	Hand-worked result
1	Introduction to Regression Analysis	The model is a dot product; always beat a baseline	RMSE 23.8 on three houses
2	Simple Linear Regression	Derive slope and intercept from two derivatives	$\hat{y} = 0.6x + 2.2$
3	Multiple Linear Regression	The Normal Equation; interpreting coefficients	$\boldsymbol{\theta} = [4.25, 2.25, 1.25]$
4	Polynomial Regression	Curves from a linear model, and how to pick the degree	$\hat{y} = 1 - 0.2x + 2x^2$
5	Cost Functions - MSE	What you optimise, and what that choice costs	MSE 0.48, MAE 0.64
6	Gradient Descent Explained	Iterate to the minimum; learning-rate stability	Two steps: 17.20 → 6.27 → 2.55
7	Regularization - Ridge and Lasso	Shrink coefficients; L1 selects, L2 does not	Lasso at $\lambda{=}10$ : $[1.0, 0]$
8	Metrics - R² and Adjusted R²	Report honestly; know when a number lies	$R^2 = 0.60$ , adjusted 0.467

The path through

diagram Diagram mermaid

  flowchart TD
  A["Introduction
model, baseline, assumptions"] --> B["Simple Linear
one feature, closed form"]
  B --> C["Multiple Linear
Normal Equation, collinearity"]
  C --> D["Polynomial
curves, degree selection"]
  C --> E["Cost Functions
MSE, MAE, Huber"]
  E --> F["Gradient Descent
iterate when closed form is too costly"]
  D --> G["Regularization
Ridge, Lasso, Elastic Net"]
  F --> G
  G --> H["Metrics
R², adjusted R², what to report"]
  H --> I["Phase 4
Classification"]

Two threads run in parallel and meet at regularisation. The modelling thread goes simple → multiple → polynomial, adding capacity. The optimisation thread goes cost function → gradient descent, explaining how any of it is computed. Regularisation needs both: it modifies the cost, and it exists because extra capacity overfits.

Before you start

You will get much more from this phase if you already have:

NumPy array basics — indexing, broadcasting, @@ for matrix multiplication
Phase 2 — train/test splits, scaling, and pipelines
Derivatives — Differentiation of Univariate Functions and Partial Differentiation and Gradients
Matrices — Matrices and Orthogonal Projections, which is what least squares geometrically is

None of these is a hard prerequisite. Every derivation on these pages is worked out in full, and every result is checked against NumPy or scikit-learn.

What you’ll be able to do afterwards

By the end of this phase you should be able to:

Derive the least-squares solution for one feature from scratch, and explain why the fitted line passes through the centroid.
Write the Normal Equation, say why lstsqlstsq is preferred over matrix inversion, and diagnose multicollinearity with a variance inflation factor.
Choose between MSE, MAE and Huber based on what your errors actually cost.
Pick a learning rate deliberately — including computing the value above which descent diverges.
Explain, geometrically, why Lasso produces exact zeros and Ridge does not.
Report a model with RMSE and $R^2$ on held-out data, and recognise when either is misleading.

How long it takes

Activity	Time
Reading the eight pages	3–4 hours
Working the hand examples with pen and paper	2 hours
Running the code and the 40 exercises	3–4 hours
The practice project below	3–5 hours
Total	11–15 hours

The hand examples are the part people skip and the part that makes the difference. All of them use small integers deliberately — you can do every one on the back of an envelope.

Practice project

Predict California housing prices, using sklearn.datasets.fetch_california_housingsklearn.datasets.fetch_california_housing (20,640 rows, 8 features). A sensible progression:

Establish the DummyRegressor(strategy="mean")DummyRegressor(strategy="mean") baseline. Write the number down.
Fit plain LinearRegressionLinearRegression. Report RMSE and $R^2$ on a held-out split.
Plot residuals against fitted values. The curvature you find is the reason for step 4.
Add PolynomialFeatures(2)PolynomialFeatures(2) inside a pipeline with StandardScalerStandardScaler. Compare cross-validated scores against step 2.
Add RidgeCVRidgeCV and LassoCVLassoCV. Check how many features Lasso keeps.
Write three sentences that a non-technical colleague could act on.

Everything you need is covered across the eight pages, and step 6 is not optional — a model nobody can act on is not finished.

quizCheck yourself

Which page should you read before Gradient Descent?
Show answer
B — Cost Functions, because gradient descent minimises the cost that page defines — Gradient descent walks downhill on a surface. Cost Functions defines that surface, proves it is convex, and explains what its gradient looks like.
What single dataset is hand-worked across this entire phase?
Show answer
B — A five-point ad-spend and sales set chosen so every intermediate value is exact — x = 1..5 and y = 2, 4, 5, 4, 5. Slope 0.6, intercept 2.2, SSE 2.4, R-squared 0.6 — all exact, all reused, so results stay comparable from page to page.
Why does this phase matter beyond regression itself?
Show answer
B — Cost functions, gradients, regularisation and held-out evaluation carry over unchanged to classification, ensembles and neural networks — Swap squared error for log loss and almost everything here applies to logistic regression; add layers and it applies to deep learning. The optimisation and evaluation machinery is universal.

Start with Introduction to Regression Analysis — the vocabulary, the general model, and the baseline you will compare everything against.

If this helped you, consider buying me a coffee ☕

Buy me a coffee