What you’ll learn

• hard against soft voting, and a case where they disagree — computed by hand
• why soft voting requires calibrated probabilities to be worth using
• stacking: letting a second-level model learn the weights instead of fixing them
• how the blender is trained on out-of-fold predictions, and why anything else leaks
• a measured comparison where none of the three combinations beats the best member
• when to reach for each, and when to ship the single model

Intuition

Bagging and boosting both build many copies of one kind of model. Voting and stacking do something different: they take models you already have — a logistic regression, a forest, a boosted tree, a KNN — and combine them.

The diversity is free. Those families make genuinely different assumptions and fail in genuinely different ways, which is exactly the condition Condorcet’s theorem requires.

The two approaches differ in one respect: who decides how much each member counts.

• Voting — you decide, in advance. Equal weights, or weights you supply.
• Stacking — a model learns them from data.

  flowchart TD
  X["Input x"] --> M1["logistic regression"]
  X --> M2["random forest"]
  X --> M3["gradient boosting"]
  X --> M4["KNN"]
  M1 --> V{"combine"}
  M2 --> V
  M3 --> V
  M4 --> V
  V -->|"voting: fixed rule"| P1["majority or mean probability"]
  V -->|"stacking: learned"| P2["a blender trained on
out-of-fold predictions"]

Hard against soft voting

Hard voting counts labels. Soft voting averages probabilities and then thresholds. They are not the same rule, and the difference matters most when confidence is unevenly distributed.

Worked example by hand

Three classifiers on one instance, predicting $P(\text{class } 1)$:

Hard voting. Two votes for class 0, one for class 1 → predict class 0.

Soft voting. Average the probabilities:

Opposite answers from identical inputs. Hard voting heard two mild “probably not”s outvote one emphatic “almost certainly”. Soft voting let the confident model carry the weight its confidence implies.

Which is right? Only if the probabilities are trustworthy. If the boosted model is systematically overconfident — and untuned boosted trees frequently are — then 0.93 is not really 0.93, and soft voting has been misled by a number that does not mean what it says.

Stacking

Voting fixes the combination rule. Stacking learns it: fit the members, collect their predictions, and train a second model — the blender, or meta-learner — to map those predictions to the final answer.

The blender can learn things no fixed rule can express: that the forest is reliable except when the KNN strongly disagrees, or that logistic regression should simply be ignored.

Training the blender without cheating

Here is the trap. If you train the members on the training set and then ask them to predict that same training set, their predictions will be far too good — a fitted random forest predicts its own training rows almost perfectly. The blender learns “always trust the forest”, and then meets a forest that is much worse on new data.

The fix is out-of-fold prediction:

  flowchart TD
  A["Training data"] --> B["Split into k folds"]
  B --> C["For each fold:
train members on the other k-1,
predict this fold"]
  C --> D["Out-of-fold predictions
one row per training row"]
  D --> E["Train the blender on those"]
  A --> F["Refit members on all the data
for use at prediction time"]

Every prediction the blender trains on comes from a model that did not see that row. Its view of each member’s reliability therefore matches what it will encounter in production. StackingClassifierStackingClassifier does this internally with its cvcv parameter; it is the entire reason the class exists rather than a five-line loop.

Measured

figureFour members, three combinations, identical foldsmatplotlib

Reading the plot

1. Hard voting was the worst combination, 1.6 points below its best member. Two weak members (0.856 and 0.858) get the same vote as KNN at 0.902 and can outvote it.
2. Soft voting recovered to 0.9000 by weighting with confidence, which is what soft voting is for.
3. Stacking matched soft voting but did not beat it. In principle the blender could learn to discard the weak members entirely; with 500 rows and 4 features of meta-data it does not have enough signal to.
4. Nothing beat KNN alone. Report this honestly and ship the single model.

The condition for stacking to pay is that the members be both reasonably strong and genuinely complementary. Here they were complementary but two of them were markedly weaker, and that turned out to be the binding constraint.

See it move

Five classifiers report a probability for each incoming instance. The two tallies show what hard and soft voting each conclude — and how often they disagree.

In code

from sklearn.datasets import make_moons
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
 
X, y = make_moons(n_samples=500, noise=0.3, random_state=1)
 
members = [
    ("logistic", make_pipeline(StandardScaler(), LogisticRegression())),
    ("tree", DecisionTreeClassifier(max_depth=4, random_state=0)),
    ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier(15))),
    ("nb", GaussianNB()),
]
 
for name, model in members:
    print(f"{name:<10} {cross_val_score(model, X, y, cv=5).mean():.4f}")
 
hard = VotingClassifier(members, voting="hard")
soft = VotingClassifier(members, voting="soft")
print(f"{'hard':<10} {cross_val_score(hard, X, y, cv=5).mean():.4f}")   # 0.8860
print(f"{'soft':<10} {cross_val_score(soft, X, y, cv=5).mean():.4f}")   # 0.9000
 
# Unequal weights, when you know one member is better
weighted = VotingClassifier(members, voting="soft", weights=[1, 2, 3, 1])

from sklearn.datasets import make_moons
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
 
X, y = make_moons(n_samples=500, noise=0.3, random_state=1)
 
members = [
    ("logistic", make_pipeline(StandardScaler(), LogisticRegression())),
    ("tree", DecisionTreeClassifier(max_depth=4, random_state=0)),
    ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier(15))),
    ("nb", GaussianNB()),
]
 
for name, model in members:
    print(f"{name:<10} {cross_val_score(model, X, y, cv=5).mean():.4f}")
 
hard = VotingClassifier(members, voting="hard")
soft = VotingClassifier(members, voting="soft")
print(f"{'hard':<10} {cross_val_score(hard, X, y, cv=5).mean():.4f}")   # 0.8860
print(f"{'soft':<10} {cross_val_score(soft, X, y, cv=5).mean():.4f}")   # 0.9000
 
# Unequal weights, when you know one member is better
weighted = VotingClassifier(members, voting="soft", weights=[1, 2, 3, 1])

from sklearn.ensemble import StackingClassifier
 
stack = StackingClassifier(
    estimators=members,
    final_estimator=LogisticRegression(),   # keep the blender simple
    cv=5,                                   # out-of-fold predictions; do not skip
    passthrough=False,                      # True also gives the blender the raw features
)
 
print(f"stacking {cross_val_score(stack, X, y, cv=5).mean():.4f}")      # 0.9000
 
stack.fit(X, y)
for name, coef in zip([n for n, _ in members], stack.final_estimator_.coef_[0]):
    print(f"  blender weight for {name:<10} {coef:+.4f}")

from sklearn.ensemble import StackingClassifier
 
stack = StackingClassifier(
    estimators=members,
    final_estimator=LogisticRegression(),   # keep the blender simple
    cv=5,                                   # out-of-fold predictions; do not skip
    passthrough=False,                      # True also gives the blender the raw features
)
 
print(f"stacking {cross_val_score(stack, X, y, cv=5).mean():.4f}")      # 0.9000
 
stack.fit(X, y)
for name, coef in zip([n for n, _ in members], stack.final_estimator_.coef_[0]):
    print(f"  blender weight for {name:<10} {coef:+.4f}")

Reading the blender’s coefficients is the payoff over voting: it tells you, in numbers, how much the meta-model decided to trust each member.

Pitfalls

Compare

Voting is the cheapest ensemble in this phase: if you already have three tuned models, it is one line and no new training.

🧪 Try It Yourself

Exercise 1 – Hard and soft disagree

Exercise 2 – Build a voting ensemble

Exercise 3 – Read the blender’s weights

Exercise 4 – Why out-of-fold matters

Exercise 5 – Did the ensemble actually earn its place?

Recap

• Hard voting counts labels; soft voting averages probabilities. On 0.45, 0.48 and 0.93 they give opposite answers.
• Soft voting is only as good as the members’ calibration — an overconfident member effectively gets extra votes.
• Stacking learns the combination weights instead of fixing them, and its coefficients tell you how much each member was trusted.
• The blender must be trained on out-of-fold predictions. A fitted forest scores 1.0 on its own training rows and 0.88 on unseen ones.
• Keep the blender simple; it sees only one number per member.
• Measured here: hard 0.8860, soft 0.9000, stacking 0.9000, best member 0.9020. No combination won. Always run that comparison.

Exercise 6 – Hard voting against soft voting on the same probabilities

Next

Phase 5 ends here. Continue to Phase 6 - Unsupervised Learning — the same rigour applied to problems with no labels at all, where even “is this good?” needs a new answer.