What you’ll learn

• how boosting differs from bagging: learners trained one after another, not in parallel
• how AdaBoost re-weights training instances to force new learners to focus on hard cases
• the predictor-weight formula Géron walks through in Chapter 7, and why a predictor that is wrong more often than not gets a negative weight
• how scikit-learn’s AdaBoostClassifierAdaBoostClassifier implements all of this in a few lines
• a live comparison: one decision stump vs. 200 boosted stumps on the same data

What boosting is

Bagging trains many models in parallel on random subsets and lets them vote. Boosting takes a different approach: it trains models sequentially, and each new model pays extra attention to the examples the previous models got wrong.

Géron calls this “hypothesis boosting” — the general idea behind every boosting method is to combine several weak learners (models barely better than a coin flip) into one strong learner, by chaining them so each one corrects its predecessor.

  flowchart LR
  D["Training data (equal weights)"] --> M1["Learner 1"]
  M1 --> W1["Reweight: boost misclassified points"]
  W1 --> M2["Learner 2 (focuses on hard points)"]
  M2 --> W2["Reweight again"]
  W2 --> M3["Learner 3"]
  M1 -- "weight α1" --> C["Weighted vote"]
  M2 -- "weight α2" --> C
  M3 -- "weight α3" --> C
  C --> P["Final prediction"]

AdaBoost intuition

AdaBoost (short for Adaptive Boosting) works like this:

1. Every training instance starts with the same weight, 1/m1/m.
2. Train a base classifier (often a shallow tree, or even just a decision stump — a tree with max_depth=1max_depth=1) and measure its weighted error rate.
3. Give that classifier a predictor weight: the more accurate it is, the higher its weight; a classifier that’s no better than random guessing gets a weight near zero, and one that’s worse than random guessing gets a negative weight.
4. Boost the weights of the misclassified instances, so the next classifier is forced to work harder on them.
5. Repeat steps 2-4 until you hit n_estimatorsn_estimators, or a perfect predictor is found.

To predict, AdaBoost doesn’t just average — it takes a weighted vote across all learners, using each learner’s predictor weight from step 3.

The predictor-weight formula

Géron’s Equation 7-2 is worth internalizing, because it explains why boosting works. For a predictor with weighted error rate rr, its predictor weight (often called alphaalpha, αα) is:

alpha = log((1 - r) / r)

alpha = log((1 - r) / r)

• if rr is small (the predictor is accurate), alphaalpha is large — it gets a big say in the final vote
• if r = 0.5r = 0.5 (pure random guessing), alpha = 0alpha = 0 — it’s ignored entirely
• if r > 0.5r > 0.5 (worse than random!), alphaalpha goes negative — the ensemble literally flips that learner’s vote

import numpy as np
 
for r in (0.05, 0.2, 0.3, 0.49):
    alpha = np.log((1 - r) / r)
    print(f"error rate {r:>4} -> predictor weight {alpha:.3f}")

import numpy as np
 
for r in (0.05, 0.2, 0.3, 0.49):
    alpha = np.log((1 - r) / r)
    print(f"error rate {r:>4} -> predictor weight {alpha:.3f}")

error rate 0.05 -> predictor weight 2.944
error rate  0.2 -> predictor weight 1.386
error rate  0.3 -> predictor weight 0.847
error rate 0.49 -> predictor weight 0.040

error rate 0.05 -> predictor weight 2.944
error rate  0.2 -> predictor weight 1.386
error rate  0.3 -> predictor weight 0.847
error rate 0.49 -> predictor weight 0.040

Every misclassified instance then has its weight multiplied by exp(alpha)exp(alpha) before the weights get renormalized — the more trustworthy the predictor, the harder it boosts the points it got wrong.

Scikit-learn example

Scikit-learn’s AdaBoostClassifierAdaBoostClassifier (and AdaBoostRegressorAdaBoostRegressor for regression) hides all of this bookkeeping behind .fit().fit(). The default base estimator is already a decision stump (max_depth=1max_depth=1), and modern scikit-learn always uses the probability-aware “SAMME” algorithm internally — no algorithm=algorithm= argument needed anymore.

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
 
X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
 
# a single decision stump on its own
stump = DecisionTreeClassifier(max_depth=1, random_state=42)
stump.fit(X_train, y_train)
print("single stump accuracy:", round(accuracy_score(y_test, stump.predict(X_test)), 3))
 
# 200 stumps, boosted
ada_clf = AdaBoostClassifier(
    estimator=DecisionTreeClassifier(max_depth=1, random_state=42),
    n_estimators=200,
    learning_rate=0.5,
    random_state=42,
)
ada_clf.fit(X_train, y_train)
print("AdaBoost accuracy:   ", round(accuracy_score(y_test, ada_clf.predict(X_test)), 3))

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
 
X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
 
# a single decision stump on its own
stump = DecisionTreeClassifier(max_depth=1, random_state=42)
stump.fit(X_train, y_train)
print("single stump accuracy:", round(accuracy_score(y_test, stump.predict(X_test)), 3))
 
# 200 stumps, boosted
ada_clf = AdaBoostClassifier(
    estimator=DecisionTreeClassifier(max_depth=1, random_state=42),
    n_estimators=200,
    learning_rate=0.5,
    random_state=42,
)
ada_clf.fit(X_train, y_train)
print("AdaBoost accuracy:   ", round(accuracy_score(y_test, ada_clf.predict(X_test)), 3))

single stump accuracy: 0.824
AdaBoost accuracy:    0.896

single stump accuracy: 0.824
AdaBoost accuracy:    0.896

200 weak stumps, chained together and reweighted, beat the single stump by a wide margin — none of them alone is a strong model, but the sequence is.

Watch the weights shift

Every round, the points a learner gets wrong grow heavier (bigger circles below), and the points it already handles well shrink back down — until the next round reshuffles who’s “hard.”

Pros and cons

Pros:

• often reaches strong accuracy with very simple base learners
• few hyperparameters to tune (n_estimatorsn_estimators, learning_ratelearning_rate, base estimator)

Cons:

• sensitive to noisy labels and outliers — a mislabeled point just keeps getting boosted, round after round
• can’t be parallelized like bagging or Random Forests

🧪 Try It Yourself

Exercise 1 – Fit an AdaBoost Classifier

Exercise 2 – Predictor Weight from Error Rate

Exercise 3 – A Weighted Majority Vote

Next

Continue to Gradient Boosting (XGBoost, LightGBM, CatBoost) — another boosting method, but instead of reweighting instances, each new model fits the residual errors left by the ensemble so far.