What you’ll learn

• boosting as the mirror image of bagging: sequential and aimed at bias, not variance
• the AdaBoost algorithm in five steps
• the predictor weight $\alpha = \frac{1}{2}\ln\frac{1-\varepsilon}{\varepsilon}$, derived and interpreted
• a complete round of AdaBoost computed by hand, weights and all
• why a learner at exactly 50% gets no vote, and one below 50% gets an inverted one
• why boosting is sensitive to noise where bagging is not

Intuition

Bagging trains many strong models independently and averages away their variance. Boosting does the opposite in every respect: it trains weak models one at a time, each one focused on the examples the previous ones got wrong, and attacks bias.

The mechanism is reweighting. After each round, misclassified examples get heavier and correctly classified ones get lighter. The next learner, fitted on those weights, is forced to pay attention to precisely the cases the committee currently fails on.

The remarkable claim — proved by Freund and Schapire in 1995 — is that any learner reliably better than chance can be boosted into an arbitrarily accurate one. A decision stump, a single split on a single feature, is enough.

  flowchart LR
  W1["equal weights"] --> S1["stump 1"]
  S1 --> E1["error → alpha 1"]
  E1 --> W2["reweight:
mistakes get heavier"]
  W2 --> S2["stump 2"]
  S2 --> E2["error → alpha 2"]
  E2 --> W3["reweight again"]
  W3 --> S3["stump m"]
  S1 --> V["weighted vote"]
  S2 --> V
  S3 --> V

Note the shape: every arrow is sequential. Stump 2 cannot start until stump 1’s errors are known, which is why boosting does not parallelise across members and a random forest does.

The math

The algorithm

1. Initialise every sample weight equally: $w_i = 1/m$.
2. Fit a weak learner on the weighted data.
3. Measure its weighted error rate:

4. Weight the predictor:

5. Reweight the samples, then renormalise so they sum to 1:

Repeat from step 2. The final prediction is the weighted vote:

Reading the alpha formula

figurePredictor weight against error ratematplotlib

Three properties fall out of the logarithm:

1. $\varepsilon = 0.5$ gives exactly zero. $\ln(1) = 0$. A learner no better than chance contributes nothing, automatically — no special case in the code.
2. The curve is antisymmetric about 0.5. A learner at 0.3 and one at 0.7 get weights of equal size and opposite sign. Being reliably wrong is exactly as useful as being reliably right, once you flip it.
3. $\alpha \to \infty$ as $\varepsilon \to 0$. A perfect learner would dominate the vote — which is why AdaBoost stops early if a learner reaches zero error, and why weak learners are the point.

Worked example by hand

Five samples, uniform initial weights of $0.2$. Round 1’s stump misclassifies samples 2 and 4.

Step 1 — the weighted error.

Step 2 — the predictor weight.

Step 3 — the multipliers.

Step 4 — apply and renormalise.

Step 5 — read the result. The two misclassified samples went from 0.200 to 0.250, a 25% increase; the three correct ones fell to 0.1667. Their combined weight is now 0.50, up from 0.40 — the next stump faces a problem where half the mass is the cases the first stump could not handle.

Notice too that the mistakes’ total weight is now exactly $0.5$. That is not a coincidence: the reweighting is constructed so the previous learner scores exactly 50% — and therefore $\alpha = 0$ — on the new distribution. Every round starts from a problem its predecessor finds impossible.

figureFour rounds of reweightingmatplotlib

See it move

The same loop, one round per beat. Watch the two mislabelled points in the middle swell until the stumps have no choice but to attend to them.

In code

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import AdaBoostClassifier
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier
 
X, y = load_breast_cancer(return_X_y=True)
 
for n_estimators in (1, 10, 50, 200):
    model = AdaBoostClassifier(
        DecisionTreeClassifier(max_depth=1),      # a stump: one split, one feature
        n_estimators=n_estimators,
        learning_rate=1.0,
        random_state=0,
    )
    print(f"{n_estimators:>3} stumps  CV {cross_val_score(model, X, y, cv=5).mean():.4f}")
 
#   1 stumps  CV 0.8998
#  10 stumps  CV 0.9385
#  50 stumps  CV 0.9666
# 200 stumps  CV 0.9772

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import AdaBoostClassifier
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier
 
X, y = load_breast_cancer(return_X_y=True)
 
for n_estimators in (1, 10, 50, 200):
    model = AdaBoostClassifier(
        DecisionTreeClassifier(max_depth=1),      # a stump: one split, one feature
        n_estimators=n_estimators,
        learning_rate=1.0,
        random_state=0,
    )
    print(f"{n_estimators:>3} stumps  CV {cross_val_score(model, X, y, cv=5).mean():.4f}")
 
#   1 stumps  CV 0.8998
#  10 stumps  CV 0.9385
#  50 stumps  CV 0.9666
# 200 stumps  CV 0.9772

One stump reaches 0.8998 — a single threshold on a single feature. Two hundred of them reach 0.9772, competitive with anything else in this curriculum on this dataset. That is the boosting claim, demonstrated: weak learners, chained, become strong.

Note the shape of the gains: 0.900 → 0.939 → 0.967 → 0.977. Each quadrupling of the budget buys roughly half the remaining error, which is the diminishing return you should expect.

Reading the weights back out

import numpy as np
 
model = AdaBoostClassifier(DecisionTreeClassifier(max_depth=1),
                           n_estimators=50, random_state=0).fit(X, y)
 
print("first five alphas:", model.estimator_weights_[:5].round(4))
print("first five errors:", model.estimator_errors_[:5].round(4))
 
# alpha should equal 0.5 * ln((1 - eps) / eps) for each stump
eps = model.estimator_errors_
manual = 0.5 * np.log((1 - eps) / eps)
print("formula matches:", np.allclose(manual, model.estimator_weights_ / 2, atol=1e-6))

import numpy as np
 
model = AdaBoostClassifier(DecisionTreeClassifier(max_depth=1),
                           n_estimators=50, random_state=0).fit(X, y)
 
print("first five alphas:", model.estimator_weights_[:5].round(4))
print("first five errors:", model.estimator_errors_[:5].round(4))
 
# alpha should equal 0.5 * ln((1 - eps) / eps) for each stump
eps = model.estimator_errors_
manual = 0.5 * np.log((1 - eps) / eps)
print("formula matches:", np.allclose(manual, model.estimator_weights_ / 2, atol=1e-6))

scikit-learn stores $2\alpha$ rather than $\alpha$ (a harmless convention difference), which is worth knowing before you compare against the formula and conclude something is broken.

learning_rate: the second brake

The $\eta$ in step 4 scales every $\alpha$:

A smaller $\eta$ makes each learner contribute less, so more of them are needed — the same shrinkage trade as in gradient boosting. learning_ratelearning_rate and n_estimatorsn_estimators must be tuned together, and their product is roughly conserved.

Where boosting breaks

Bagging is robust to label noise: a mislabelled point ends up in about 63% of the bootstrap samples and gets outvoted. Boosting hunts it down.

A mislabelled example is misclassified every round by construction — no learner can get it right — so its weight grows exponentially. After enough rounds the ensemble is devoting most of its capacity to fitting a handful of corrupted labels.

Pitfalls

Compare

🧪 Try It Yourself

Exercise 1 – The alpha formula

Exercise 2 – One round of reweighting

Exercise 3 – Weak learners become strong

Exercise 4 – A weighted majority vote

Exercise 5 – Watch boosting chase a bad label

Recap

• Boosting is bagging’s mirror image: weak learners, trained sequentially, attacking bias.
• $\alpha = \frac{1}{2}\ln\frac{1-\varepsilon}{\varepsilon}$ gives +1.099 at 10% error, exactly 0 at 50%, and −0.424 at 70% — the vote is inverted rather than discarded.
• Hand-worked round: error 0.4, $\alpha = 0.2027$, mistake weights rise from 0.200 to 0.250 and their total mass becomes exactly 0.5.
• Each round hands its successor a problem the previous learner finds no better than chance.
• One decision stump on breast cancer scores 0.8998; two hundred score 0.9772.
• n_estimatorsn_estimators can overfit here, unlike a random forest, and learning_ratelearning_rate trades against it.
• Boosting is fragile to label noise for exactly the reason it is powerful on clean data.

Exercise 6 – Run one AdaBoost weight update by hand

Next

Continue to Gradient Boosting (XGBoost, LightGBM, CatBoost) — the same sequential idea with reweighting replaced by gradient descent in function space, and the family of libraries that dominates tabular machine learning.