The intuition: many opinions beat one

If you ask one person for a guess, you can get a bad answer.

If you ask 100 people and average their guesses, the result is often better.

Ensembles do the same for models. Géron opens Chapter 7 of Hands-On Machine Learning with this exact analogy, calling it the wisdom of the crowd: aggregate the predictions of a group of predictors — an ensemble — and you’ll often beat the best individual predictor in that group.

Why ensembles work

Ensembles improve performance through:

1) Variance reduction (bagging)

• high-variance models (like deep decision trees) can overfit
• averaging many trees reduces sensitivity to noise

2) Bias reduction (boosting)

• weak learners can underfit
• boosting adds learners that fix previous errors

3) Better decision boundaries (model diversity)

If models make different mistakes, combining helps.

  flowchart LR
  M1[Model 1] --> C[Combine]
  M2[Model 2] --> C
  M3[Model 3] --> C
  C --> P[Better prediction]

What “diversity” means

Models should not all make the same mistakes.

How diversity is created:

• different subsamples of data (bagging)
• different feature subsets (random forests)
• sequential focus on errors (boosting)
• entirely different algorithms (voting classifiers)

From the book: voting classifiers

Suppose you’ve trained a Logistic Regression classifier, an SVM, a Random Forest, and a K-Nearest Neighbors classifier, and each achieves about 80% accuracy on its own. A very simple way to build an even better classifier is to let them vote:

• hard voting — predict the class that gets the most votes across all classifiers
• soft voting — average each classifier’s predicted class probabilities, and predict whichever class has the highest average probability

Soft voting usually wins because it gives more weight to confident votes, but it requires every base classifier to expose predict_proba()predict_proba() (for SVCSVC, that means setting probability=Trueprobability=True).

from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, VotingClassifier
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score
 
X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
 
log_clf = LogisticRegression()
rnd_clf = RandomForestClassifier(n_estimators=100, random_state=42)
svm_clf = SVC()
 
voting_clf = VotingClassifier(
    estimators=[("lr", log_clf), ("rf", rnd_clf), ("svc", svm_clf)],
    voting="hard",
)
 
for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    print(clf.__class__.__name__, round(accuracy_score(y_test, y_pred), 3))

from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, VotingClassifier
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score
 
X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
 
log_clf = LogisticRegression()
rnd_clf = RandomForestClassifier(n_estimators=100, random_state=42)
svm_clf = SVC()
 
voting_clf = VotingClassifier(
    estimators=[("lr", log_clf), ("rf", rnd_clf), ("svc", svm_clf)],
    voting="hard",
)
 
for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    print(clf.__class__.__name__, round(accuracy_score(y_test, y_pred), 3))

Notice the VotingClassifierVotingClassifier line usually edges out every individual model — that’s the whole point of an ensemble.

Weak learners can build a strong learner

Here’s the surprising part: even if every individual classifier is a weak learner (barely better than a coin flip), the ensemble can still be a strong learner, as long as there are enough of them and they’re sufficiently independent.

Géron’s analogy: imagine a biased coin with a 51% chance of heads. Toss it once and you’ve learned almost nothing. Toss it 1,000 times and the probability of ending up with a majority of heads is already close to 75%. Toss it 10,000 times and that climbs above 97%. This is the law of large numbers — the more (independent) votes you gather, the closer the majority tracks the true underlying probability.

The catch: this only holds if the classifiers’ errors are uncorrelated. Classifiers trained on the same data tend to make similar mistakes, which is why diversity (different algorithms, different data subsets, different features) matters so much for real ensembles.

Visualize it

Bagging trains models in parallel and votes, while boosting trains models in sequence and corrects mistakes.

  flowchart TB
  T["Training data"] --> BG["Bagging: many models trained in PARALLEL on random subsets to majority vote"]
  T --> BO["Boosting: models trained in SEQUENCE, each fixing the previous one's errors to weighted vote"]

Below is the law of large numbers in action — ten independent series of biased-coin tosses, each converging toward the true 51% bias as more coins are tossed:

The tradeoff

Ensembles can be:

• less interpretable
• heavier (more compute)

But on many tabular problems, they are the strongest first choice.

Mini-checkpoint

Which seems more likely to generalize?

• one deep tree
• 200 trees averaged

(Usually the averaged forest.)

🧪 Try It Yourself

Exercise 1 – Build a Hard Voting Classifier

Exercise 2 – Enable Soft Voting for SVC

Exercise 3 – From Weak to Strong Learner

Next

Continue to Bagging: Random Forest Regressor/Classifier — see how sampling the training set itself (instead of switching algorithms) is another powerful way to build a diverse ensemble.