What bagging is

Bagging = Bootstrap Aggregating.

Steps:

1. sample (with replacement) multiple datasets from the training set
2. train one model per sample
3. aggregate predictions (vote/average)

  flowchart TD
  D[Training data] --> S1[Bootstrap sample 1]
  D --> S2[Bootstrap sample 2]
  D --> S3[Bootstrap sample 3]
  S1 --> T1[Tree 1]
  S2 --> T2[Tree 2]
  S3 --> T3[Tree 3]
  T1 --> A[Aggregate]
  T2 --> A
  T3 --> A
  A --> P[Final prediction]

Bagging vs. pasting

Per Hands-On Machine Learning, both bagging and pasting train the same algorithm on different random subsets of the training set — the only difference is whether sampling is done with replacement or without:

• bagging — sampling with replacement (an instance can be picked more than once by the same predictor)
• pasting — sampling without replacement

Each individual predictor ends up with higher bias than if it saw the full training set, but aggregating many of them reduces both bias and variance — the net effect is an ensemble with similar bias but noticeably lower variance than a single predictor. Because bagging’s extra resampling adds more diversity, it usually produces slightly better models than pasting, which is why it’s the default. Both scale beautifully because every predictor can be trained (and later queried) fully in parallel.

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
 
X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
 
# 500 trees, each trained on 100 instances sampled WITH replacement (bagging)
bag_clf = BaggingClassifier(
    DecisionTreeClassifier(),
    n_estimators=500,
    max_samples=100,
    bootstrap=True,     # set to False for pasting
    n_jobs=-1,
    random_state=42,
)
bag_clf.fit(X_train, y_train)
print("Test accuracy:", round(bag_clf.score(X_test, y_test), 3))

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
 
X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
 
# 500 trees, each trained on 100 instances sampled WITH replacement (bagging)
bag_clf = BaggingClassifier(
    DecisionTreeClassifier(),
    n_estimators=500,
    max_samples=100,
    bootstrap=True,     # set to False for pasting
    n_jobs=-1,
    random_state=42,
)
bag_clf.fit(X_train, y_train)
print("Test accuracy:", round(bag_clf.score(X_test, y_test), 3))

BaggingClassifierBaggingClassifier automatically switches to soft voting instead of hard voting whenever the base estimator can output class probabilities — which Decision Trees can.

Out-of-bag (OOB) evaluation

With bagging, each predictor only ever sees about 63% of the training instances on average (as m grows, this ratio approaches 1 - exp(-1) ≈ 63.2%1 - exp(-1) ≈ 63.2%). The remaining ~37% — the out-of-bag instances — are never seen by that predictor during training, so they make a free validation set. Set oob_score=Trueoob_score=True and scikit-learn will average the OOB evaluations across all predictors for you, with no need for a separate holdout set.

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_moons
 
X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
 
bag_clf = BaggingClassifier(
    DecisionTreeClassifier(),
    n_estimators=500,
    bootstrap=True,
    oob_score=True,
    n_jobs=-1,
    random_state=42,
)
bag_clf.fit(X, y)
print("OOB score:", round(bag_clf.oob_score_, 3))

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_moons
 
X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
 
bag_clf = BaggingClassifier(
    DecisionTreeClassifier(),
    n_estimators=500,
    bootstrap=True,
    oob_score=True,
    n_jobs=-1,
    random_state=42,
)
bag_clf.fit(X, y)
print("OOB score:", round(bag_clf.oob_score_, 3))

Random Patches and Random Subspaces

BaggingClassifierBaggingClassifier can also sample features, controlled by max_featuresmax_features and bootstrap_featuresbootstrap_features (they work just like max_samplesmax_samples/bootstrapbootstrap, but for columns instead of rows). This is especially useful on high-dimensional data such as images:

• sampling both instances and features → Random Patches
• keeping all instances but sampling features → Random Subspaces

Random Forest in one line

A Random Forest is bagging + decision trees + random feature selection at each split.

Instead of hand-assembling a BaggingClassifierBaggingClassifier around DecisionTreeClassifierDecisionTreeClassifier, use the dedicated, tree-optimized RandomForestClassifierRandomForestClassifier / RandomForestRegressorRandomForestRegressor classes. Random Forests add one extra trick beyond plain bagging: instead of searching every feature for the best split at each node, each tree only considers a random subset of features. This randomness increases diversity → improves generalization.

Classification vs regression

• RandomForestClassifier: majority vote
• RandomForestRegressor: average prediction

Scikit-learn examples

from sklearn.ensemble import RandomForestClassifier
 
rf = RandomForestClassifier(
    n_estimators=200,
    max_depth=None,
    random_state=42,
    n_jobs=-1,
)

from sklearn.ensemble import RandomForestClassifier
 
rf = RandomForestClassifier(
    n_estimators=200,
    max_depth=None,
    random_state=42,
    n_jobs=-1,
)

from sklearn.ensemble import RandomForestRegressor
 
rf = RandomForestRegressor(
    n_estimators=300,
    random_state=42,
    n_jobs=-1,
)

from sklearn.ensemble import RandomForestRegressor
 
rf = RandomForestRegressor(
    n_estimators=300,
    random_state=42,
    n_jobs=-1,
)

Extra-Trees

Push the randomness one step further: instead of searching for the best possible threshold for each candidate feature, pick a random threshold too. A forest built this way is called Extremely Randomized Trees, or Extra-Trees. It trades a bit more bias for lower variance, and since it skips the expensive threshold search, it also trains noticeably faster than a regular Random Forest. ExtraTreesClassifierExtraTreesClassifier / ExtraTreesRegressorExtraTreesRegressor share the exact same API as their Random Forest counterparts — the only way to know which will perform better on your data is to try both.

Feature importance

One of the best perks of Random Forests: they make it easy to rank feature importance. Scikit-learn measures how much each tree node that uses a feature reduces impurity on average (weighted by how many samples reach that node), then scales the results so they sum to 1. Access them via feature_importances_feature_importances_:

from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
 
iris = load_iris()
rnd_clf = RandomForestClassifier(n_estimators=200, random_state=42)
rnd_clf.fit(iris.data, iris.target)
 
for name, score in zip(iris.feature_names, rnd_clf.feature_importances_):
    print(name, round(score, 3))

from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
 
iris = load_iris()
rnd_clf = RandomForestClassifier(n_estimators=200, random_state=42)
rnd_clf.fit(iris.data, iris.target)
 
for name, score in zip(iris.feature_names, rnd_clf.feature_importances_):
    print(name, round(score, 3))

sepal length (cm) 0.102
sepal width (cm)  0.021
petal length (cm) 0.459
petal width (cm)  0.418

sepal length (cm) 0.102
sepal width (cm)  0.021
petal length (cm) 0.459
petal width (cm)  0.418

Petal length and width dominate — sepal measurements barely matter. This makes Random Forests a quick, handy tool for feature selection even before you commit to a final model.

Useful hyperparameters

• n_estimatorsn_estimators: more trees → better but slower
• max_depthmax_depth: limits overfitting
• min_samples_leafmin_samples_leaf: smooths leaves
• max_featuresmax_features: controls feature randomness

Visualize it

Every bootstrap sample (“bag”) is drawn with replacement from the same training set, so some instances get picked multiple times while others are skipped entirely — those skipped instances are the out-of-bag ones used for free validation:

Mini-checkpoint

First try:

• deep trees in forest
• tune max_depthmax_depth and min_samples_leafmin_samples_leaf if overfitting

🧪 Try It Yourself

Exercise 1 – Bagging vs Pasting

Exercise 2 – Free Validation with OOB Score

Exercise 3 – Rank Feature Importance

Next

Continue to Boosting: Introduction to AdaBoost — instead of training trees in parallel on random subsets, boosting trains them one after another, each one focusing harder on the mistakes of the last.