Phase 5 - Ensemble Learning

The single most reliable way to improve a tabular model is to stop using one model. Ensembles win Kaggle competitions, they are the default in production, and gradient boosting in particular is the strongest general-purpose method in this curriculum.

But they are not magic, and this phase is careful about that. One page ends with a measurement where no combination of four models beat the best single member. Knowing when combining fails is what separates using ensembles from cargo-culting them.

What this phase covers

Five pages, and the order matters: the first establishes the condition all the others must satisfy.

#	Page	Mechanism	Key measurement
1	The Power of Ensembles	Why combining works at all	60% voters → 97.9% at n=101
2	Bagging & Random Forests	Resample rows and features	bagging 0.9417 vs pasting 0.8833
3	AdaBoost	Reweight the mistakes	1 stump 0.8998 → 200 stumps 0.9772
4	Gradient Boosting	Fit the residual = descend the gradient	MSE 66.7 → 29.2 → 8.1 in two stages
5	Stacking and Voting	Learn the weights	no combination beat the best member

The one equation

Everything here is governed by the variance of an average of correlated models:

\operatorname{Var}\left(\frac{1}{n}\sum_{i=1}^{n} f_i\right) = \underbrace{\rho\,\sigma^{2}}_{\text{cannot be reduced}} + \underbrace{\frac{1-\rho}{n}\,\sigma^{2}}_{\text{shrinks with } n}

Adding members only shrinks the second term. Correlation is a floor, and every design decision in this phase is an attempt to push $\rho$ down:

Technique	How it lowers $\rho$
Bootstrap sampling	Each model sees a different 63% of the rows
Feature subsampling	Trees cannot all split on the dominant feature
Random thresholds (Extra-Trees)	Even the split points differ
Sequential reweighting	Each model faces a different distribution by construction
Different algorithms	The families disagree by design

The three families

diagram Diagram mermaid

  flowchart TD
  A["Many models"] --> B["Bagging
parallel, strong members"]
  A --> C["Boosting
sequential, weak members"]
  A --> D["Stacking
parallel, different families"]
  B --> B2["attacks VARIANCE
Random Forest, Extra-Trees"]
  C --> C2["attacks BIAS
AdaBoost, XGBoost, LightGBM"]
  D --> D2["learns the weighting
VotingClassifier, StackingClassifier"]
  B2 --> E["More members never hurt"]
  C2 --> F["More members CAN overfit"]
  D2 --> G["Only as good as its members"]

	Bagging	Boosting	Stacking
Members	Deep trees	Shallow trees or stumps	Any, ideally different families
Trained	In parallel	Sequentially	In parallel, then a blender
Attacks	Variance	Bias	Both
More members overfit	No	Yes	No, but adds no value either
Robust to label noise	Yes	No	Depends on the members
Tuning burden	Low	High	Moderate
Typical tabular result	Very good	Best	Small extra gain

Each maps onto a term of the bias-variance decomposition. That is not a coincidence or an analogy — it is the design rationale.

Before you start

Decision Trees — the base learner for four of the five pages. That page ends by noting trees are unstable; this phase is what that instability is for.
Cross-validation — every comparison here is on identical folds, and out-of-fold prediction is the core of stacking.
Bias and variance — the vocabulary the whole phase uses.

What you’ll be able to do afterwards

Explain, with the variance formula, why an ensemble of correlated models is nearly pointless.
Derive the 63.2% bootstrap coverage and use out-of-bag scoring.
Compute an AdaBoost round by hand — error, $\alpha$ , and the reweighting.
Explain why fitting the residual is gradient descent, and swap the loss to fit a new problem.
Choose between hard voting, soft voting and stacking on calibration and member-strength grounds.
Recognise when the ensemble should be abandoned in favour of a single model.

How long it takes

Activity	Time
Reading the five pages	3–4 hours
Working the hand examples	2 hours
Running the code and the 25 exercises	3–4 hours
The practice project below	4–6 hours
Total	12–16 hours

Practice project

Build a tabular leaderboard for one dataset. Pick anything with 5,000+ rows and mixed feature types, then fill in this table with 5-fold cross-validated scores on identical folds:

Model	Score	Training time	Notes
`DummyClassifierDummyClassifier`			The floor
Logistic regression			The cheap baseline — it sometimes wins
Single decision tree
Random forest			Try `max_featuresmax_features`
Extra-Trees			Usually faster, often better
AdaBoost
`HistGradientBoostingHistGradientBoosting`			Tune `learning_ratelearning_rate` with early stopping
Soft voting of the best three
Stacking of the best three

Then answer three questions in writing:

Did any ensemble beat the best single model? By how much, and is it larger than the fold-to-fold standard deviation?
What did it cost? Training time, prediction latency, and whether you could still explain a prediction.
Which would you ship? With a reason.

If the honest answer to question 1 is “no”, that is a finding, not a failure — and the page on stacking reaches exactly that answer on its own data.

quizCheck yourself

Your 200-model ensemble has average pairwise correlation 0.85. What is the main problem?
Show answer
B — Correlation puts a floor at 0.85 of a single model's variance — a handful of models would have achieved almost the same thing — Variance is rho + (1-rho)/n = 0.85 + 0.00075. Adding models shrinks only the second term, which is already negligible. The fix is diversity, not quantity.
Which ensemble family cannot be parallelised across its members?
Show answer
B — Boosting — Each boosting round depends on the previous round's errors. Bagging, voting and stacking all train their members independently.
For which family does adding more members eventually hurt?
Show answer
B — Boosting — each round fits the previous residuals, so enough rounds fit noise — A random forest's n_estimators cannot overfit. A boosted model's can, which is why early stopping is mandatory there and irrelevant in bagging.

Start with The Power of Ensembles - Why Combine Models? — Condorcet’s theorem, the correlation floor, and the condition every later page has to satisfy.

If this helped you, consider buying me a coffee ☕

Buy me a coffee