What you’ll learn

• hard voting vs. soft voting, and why soft voting usually wins
• how stacking replaces a simple vote with a trained “meta-learner”
• why the meta-learner has to be trained on held-out predictions, not the same data the base models trained on
• how to build both with scikit-learn’s VotingClassifierVotingClassifier and StackingClassifierStackingClassifier
• a live accuracy comparison across every approach, on the same dataset

Voting ensembles

Voting is the simplest way to combine several already-trained, different models: let them each cast a vote, and combine the votes.

• hard voting — predict whichever class gets the most raw votes (majority rule)
• soft voting — average each model’s predicted class probabilities, and predict whichever class has the highest average probability

  flowchart LR
  A["Logistic Regression"] --> V["Voting"]
  B["Random Forest"] --> V
  C["SVM"] --> V
  V --> P["Final prediction"]

Soft voting usually beats hard voting because it gives more weight to confident predictions instead of treating every vote equally — but every base model needs a working predict_proba()predict_proba() method. For scikit-learn’s SVCSVC, that means you must ask for it explicitly with probability=Trueprobability=True (this makes fitting slower, since it now uses internal cross-validation to calibrate those probabilities).

Stacking ensembles

Voting aggregates predictions with a fixed rule (majority, or average). Stacking (short for stacked generalization) goes one step further: instead of a fixed rule, it trains a model — called a blender, or meta-learner — to learn how to best combine the base models’ predictions.

  flowchart TD
  X["Features"] --> M1["Base model 1 (Random Forest)"]
  X --> M2["Base model 2 (SVM)"]
  X --> M3["Base model 3 (Logistic Regression)"]
  M1 --> Z["Meta-features: (p1, p2, p3)"]
  M2 --> Z
  M3 --> Z
  Z --> META["Meta-learner (blender)"]
  META --> Y["Final prediction"]

How the blender gets trained without cheating

If the blender trained on the same predictions the base models made on their own training data, it would be learning from predictions that are “too good” — the base models have already memorized some of that data. Géron’s fix is to use a hold-out set:

1. Split the training data into two parts.
2. Train the base models on the first part only.
3. Use the trained base models to predict on the second, held-out part — these predictions are “clean,” since the base models never saw those instances during training.
4. Build a new training set from those clean predictions (one column per base model) with the original targets, and train the blender on that.

It’s even possible to stack multiple layers of blenders this way — split the data into three parts instead of two, and each layer trains on the previous layer’s held-out predictions.

Scikit-learn examples

from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, VotingClassifier
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score
 
X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
 
def make_base_models():
    return [
        ("lr", LogisticRegression()),
        ("rf", RandomForestClassifier(n_estimators=100, random_state=42)),
        ("svc", SVC(probability=True, random_state=42)),
    ]
 
hard_voter = VotingClassifier(estimators=make_base_models(), voting="hard")
soft_voter = VotingClassifier(estimators=make_base_models(), voting="soft")
 
for name, clf in (("hard voting", hard_voter), ("soft voting", soft_voter)):
    clf.fit(X_train, y_train)
    acc = accuracy_score(y_test, clf.predict(X_test))
    print(name, round(acc, 3))

from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, VotingClassifier
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score
 
X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
 
def make_base_models():
    return [
        ("lr", LogisticRegression()),
        ("rf", RandomForestClassifier(n_estimators=100, random_state=42)),
        ("svc", SVC(probability=True, random_state=42)),
    ]
 
hard_voter = VotingClassifier(estimators=make_base_models(), voting="hard")
soft_voter = VotingClassifier(estimators=make_base_models(), voting="soft")
 
for name, clf in (("hard voting", hard_voter), ("soft voting", soft_voter)):
    clf.fit(X_train, y_train)
    acc = accuracy_score(y_test, clf.predict(X_test))
    print(name, round(acc, 3))

hard voting 0.912
soft voting 0.92

hard voting 0.912
soft voting 0.92

from sklearn.ensemble import StackingClassifier
 
stack_clf = StackingClassifier(
    estimators=[
        ("rf", RandomForestClassifier(n_estimators=100, random_state=42)),
        ("svc", SVC(probability=True, random_state=42)),
    ],
    final_estimator=LogisticRegression(),
)
stack_clf.fit(X_train, y_train)
print(round(accuracy_score(y_test, stack_clf.predict(X_test)), 3))

from sklearn.ensemble import StackingClassifier
 
stack_clf = StackingClassifier(
    estimators=[
        ("rf", RandomForestClassifier(n_estimators=100, random_state=42)),
        ("svc", SVC(probability=True, random_state=42)),
    ],
    final_estimator=LogisticRegression(),
)
stack_clf.fit(X_train, y_train)
print(round(accuracy_score(y_test, stack_clf.predict(X_test)), 3))

0.912

0.912

Every combination here (0.912-0.92) beats every individual base model on its own (Logistic Regression alone scores 0.864) — that’s the payoff of combining diverse, complementary models.

Visualize it: votes tallying up

Three base learners each output a probability of class 1 for the same instance. Watch each model’s bar arrive in turn, then watch the votes get tallied one model at a time — hard voting only cares which side of 0.5 each one lands on; soft voting averages the raw probabilities, and can flip the decision that hard voting would have made.

Mini-checkpoint

Try a voting ensemble with:

• one linear model (Logistic Regression)
• one non-linear model (SVM)
• one tree-based model (Random Forest)

Compare its accuracy against each base model individually, then swap voting="hard"voting="hard" for voting="soft"voting="soft" and see whether it improves further.

🧪 Try It Yourself

Exercise 1 – Build a Hard Voting Ensemble

Exercise 2 – Switch to Soft Voting

Exercise 3 – Train a Stacking Classifier

Next

That wraps up Phase 5. Continue to Introduction to Clustering in Phase 6 — you’ve spent this whole phase combining supervised models; next you’ll learn how to find structure in data that has no labels at all.