What boosting is

Boosting builds models sequentially.

Each new model focuses more on examples the previous models got wrong.

  flowchart LR
  D[Train data] --> M1[Weak learner 1]
  M1 --> E1[Errors]
  E1 --> M2[Weak learner 2 (focus on errors)]
  M2 --> E2[Errors]
  E2 --> M3[Weak learner 3]
  M1 --> C[Combine (weighted)]
  M2 --> C
  M3 --> C
  C --> P[Final prediction]

AdaBoost intuition

AdaBoost (Adaptive Boosting):

• starts with equal weight for all data points
• increases weights on misclassified points
• trains next learner to improve on those

Often uses shallow decision trees (stumps).

Scikit-learn example

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
 
base = DecisionTreeClassifier(max_depth=1, random_state=42)
ada = AdaBoostClassifier(
    estimator=base,
    n_estimators=200,
    learning_rate=0.5,
    random_state=42,
)

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
 
base = DecisionTreeClassifier(max_depth=1, random_state=42)
ada = AdaBoostClassifier(
    estimator=base,
    n_estimators=200,
    learning_rate=0.5,
    random_state=42,
)

Pros and cons

Pros:

• can achieve strong performance
• often better than a single tree

Cons:

• sensitive to noisy labels/outliers

Mini-checkpoint

If your dataset has lots of mislabeled samples, boosting may struggle.