Stacking and Voting Classifiers

What you’ll learn

• hard against soft voting, and a case where they disagree — computed by hand
• why soft voting requires calibrated probabilities to be worth using
• stacking: letting a second-level model learn the weights instead of fixing them
• how the blender is trained on out-of-fold predictions, and why anything else leaks
• a measured comparison where none of the three combinations beats the best member
• when to reach for each, and when to ship the single model

Intuition

Bagging and boosting both build many copies of one kind of model. Voting and stacking do something different: they take models you already have — a logistic regression, a forest, a boosted tree, a KNN — and combine them.

The diversity is free. Those families make genuinely different assumptions and fail in genuinely different ways, which is exactly the condition Condorcet’s theorem requires.

The two approaches differ in one respect: who decides how much each member counts.

• Voting — you decide, in advance. Equal weights, or weights you supply.
• Stacking — a model learns them from data.

diagram Diagram mermaid

  flowchart TD
  X["Input x"] --> M1["logistic regression"]
  X --> M2["random forest"]
  X --> M3["gradient boosting"]
  X --> M4["KNN"]
  M1 --> V{"combine"}
  M2 --> V
  M3 --> V
  M4 --> V
  V -->|"voting: fixed rule"| P1["majority or mean probability"]
  V -->|"stacking: learned"| P2["a blender trained on
out-of-fold predictions"]

Hard against soft voting

Hard voting counts labels. Soft voting averages probabilities and then thresholds. They are not the same rule, and the difference matters most when confidence is unevenly distributed.

$$\text{hard: } \hat{y} = \operatorname*{arg\,max}_{k}\sum_{j=1}^{J}\mathbb{1}\!\left[\hat{y}_j = k\right] \qquad \text{soft: } \hat{y} = \operatorname*{arg\,max}_{k}\frac{1}{J}\sum_{j=1}^{J}\hat{p}_{jk}$$

Worked example by hand

Three classifiers on one instance, predicting $P(\text{class } 1)$:

Classifier	$P(\text{class }1)$	Hard vote
logistic regression	0.45	class 0
decision tree	0.48	class 0
gradient boosting	0.93	class 1

Hard voting. Two votes for class 0, one for class 1 → predict class 0.

Soft voting. Average the probabilities:

$$\bar{p} = \frac{0.45 + 0.48 + 0.93}{3} = \frac{1.86}{3} = 0.62 \;\Rightarrow\; \textbf{predict class 1}$$

Opposite answers from identical inputs. Hard voting heard two mild “probably not”s outvote one emphatic “almost certainly”. Soft voting let the confident model carry the weight its confidence implies.

Which is right? Only if the probabilities are trustworthy. If the boosted model is systematically overconfident — and untuned boosted trees frequently are — then 0.93 is not really 0.93, and soft voting has been misled by a number that does not mean what it says.

Soft voting is only as good as the calibration

Logistic regression is usually well calibrated. Naive Bayes is notoriously not, routinely emitting 0.9999. SVMs produce probabilities only via Platt scaling. Wrap poorly-calibrated members in CalibratedClassifierCV before soft voting, or use hard voting instead.

	Hard voting	Soft voting
Uses	Predicted labels	Predicted probabilities
Needs predict_proba	No	Yes
Sensitive to calibration	No	Very
Confident members	Count the same as unsure ones	Count more
Usually better when	Members are poorly calibrated	Members are well calibrated

Stacking

Voting fixes the combination rule. Stacking learns it: fit the members, collect their predictions, and train a second model — the blender, or meta-learner — to map those predictions to the final answer.

The blender can learn things no fixed rule can express: that the forest is reliable except when the KNN strongly disagrees, or that logistic regression should simply be ignored.

Training the blender without cheating

Here is the trap. If you train the members on the training set and then ask them to predict that same training set, their predictions will be far too good — a fitted random forest predicts its own training rows almost perfectly. The blender learns “always trust the forest”, and then meets a forest that is much worse on new data.

The fix is out-of-fold prediction:

diagram Diagram mermaid

  flowchart TD
  A["Training data"] --> B["Split into k folds"]
  B --> C["For each fold:
train members on the other k-1,
predict this fold"]
  C --> D["Out-of-fold predictions
one row per training row"]
  D --> E["Train the blender on those"]
  A --> F["Refit members on all the data
for use at prediction time"]

Every prediction the blender trains on comes from a model that did not see that row. Its view of each member’s reliability therefore matches what it will encounter in production. StackingClassifier does this internally with its cv parameter; it is the entire reason the class exists rather than a five-line loop.

Measured

figure Four members, three combinations, identical folds matplotlib

KNN alone reaches 0.9020. Hard voting drops to 0.8860; soft voting and stacking both reach 0.9000. Not one combination beat the best member.

Model	5-fold CV accuracy
logistic regression	0.8560
decision tree (depth 4)	0.9000
KNN (k=15)	0.9020
Gaussian naive Bayes	0.8580
hard vote	0.8860
soft vote	0.9000
stacking (logistic blender)	0.9000

Reading the plot

1. Hard voting was the worst combination, 1.6 points below its best member. Two weak members (0.856 and 0.858) get the same vote as KNN at 0.902 and can outvote it.
2. Soft voting recovered to 0.9000 by weighting with confidence, which is what soft voting is for.
3. Stacking matched soft voting but did not beat it. In principle the blender could learn to discard the weak members entirely; with 500 rows and 4 features of meta-data it does not have enough signal to.
4. Nothing beat KNN alone. Report this honestly and ship the single model.

The condition for stacking to pay is that the members be both reasonably strong and genuinely complementary. Here they were complementary but two of them were markedly weaker, and that turned out to be the binding constraint.

See it move

Five classifiers report a probability for each incoming instance. The two tallies show what hard and soft voting each conclude — and how often they disagree.

sketch Hard and soft voting, side by side p5.js Pause Restart

Five members report probabilities for one instance. The bars show each member's confidence; the two verdicts below show what counting labels and averaging probabilities each decide.

In code

voting.py

from sklearn.datasets import make_moons
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
 
X, y = make_moons(n_samples=500, noise=0.3, random_state=1)
 
members = [
    ("logistic", make_pipeline(StandardScaler(), LogisticRegression())),
    ("tree", DecisionTreeClassifier(max_depth=4, random_state=0)),
    ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier(15))),
    ("nb", GaussianNB()),
]
 
for name, model in members:
    print(f"{name:<10} {cross_val_score(model, X, y, cv=5).mean():.4f}")
 
hard = VotingClassifier(members, voting="hard")
soft = VotingClassifier(members, voting="soft")
print(f"{'hard':<10} {cross_val_score(hard, X, y, cv=5).mean():.4f}")   # 0.8860
print(f"{'soft':<10} {cross_val_score(soft, X, y, cv=5).mean():.4f}")   # 0.9000
 
# Unequal weights, when you know one member is better
weighted = VotingClassifier(members, voting="soft", weights=[1, 2, 3, 1])

stacking.py

from sklearn.ensemble import StackingClassifier
 
stack = StackingClassifier(
    estimators=members,
    final_estimator=LogisticRegression(),   # keep the blender simple
    cv=5,                                   # out-of-fold predictions; do not skip
    passthrough=False,                      # True also gives the blender the raw features
)
 
print(f"stacking {cross_val_score(stack, X, y, cv=5).mean():.4f}")      # 0.9000
 
stack.fit(X, y)
for name, coef in zip([n for n, _ in members], stack.final_estimator_.coef_[0]):
    print(f"  blender weight for {name:<10} {coef:+.4f}")

Reading the blender’s coefficients is the payoff over voting: it tells you, in numbers, how much the meta-model decided to trust each member.

Keep the blender simple

A logistic regression or a shallow tree. The blender sees only $J$ numbers per row — with four members that is four features — so a complex blender overfits immediately. passthrough=True gives it the original features too, which occasionally helps and often overfits.

pch.algoTag Voting and Stacking Supervised · Classification and Regression · Heterogeneous ensemble

pch.algoApi sklearn.ensemble.VotingClassifier / StackingClassifier

pch.algoAssumes

• Members make genuinely different errors — different families, not different seeds
• For soft voting, member probabilities are calibrated
• For stacking, enough training data for the blender to learn from J numbers per row

pch.algoCost

pch.algoTrain

sum of member costs; stacking multiplies by k folds

pch.algoPredict

sum of member costs

pch.algoMemory

sum of member sizes

J = number of members, k = folds used for out-of-fold predictions

pch.algoHyperparams

• voting default hard 'soft' averages probabilities and usually wins when members are calibrated. Check calibration first.
• weights default None Manual per-member weights for voting. A cheap alternative to stacking when you already know the ranking.
• final_estimator default LogisticRegression The blender. Keep it simple; it only sees J features.
• cv default 5 Folds for out-of-fold predictions. Lower is faster and leakier; do not set it to 'prefit' unless you know exactly why.
• passthrough default False Give the blender the original features as well. Occasionally helps, frequently overfits.

pch.algoReachFor

• You already have several tuned models of different families
• The members' errors are genuinely uncorrelated
• The last fraction of a point matters — this is standard practice in competitions
• Inference cost is not a constraint

pch.algoLookElsewhere

• Members are all the same family — bagging or boosting is the right tool
• Prediction latency matters; every member runs on every request
• One member clearly dominates the rest, as measured above
• The training set is small; the blender needs data too

Pitfalls

Training the blender on in-sample predictions

A fitted forest predicts its own training rows nearly perfectly. A blender trained on those learns to trust it absolutely, and then meets a forest that is much worse on new data. Always use out-of-fold predictions — which is what cv= does for you.

Soft voting with uncalibrated members

Naive Bayes routinely emits 0.9999 and does not mean it. Averaging that against an honest 0.6 hands the decision to the least trustworthy number in the ensemble. Calibrate or use hard voting.

A blender that is too complex

With four members the blender sees four features. A random forest blender will memorise the out-of-fold predictions and add nothing. Logistic regression or a depth-2 tree.

Not comparing against the best member

The measurement above found no combination beating KNN alone. Run that comparison every time, on identical folds, and be prepared to ship the single model.

Compare

	Bagging	Boosting	Voting	Stacking
Members	One family, resampled	One family, sequential	Different families	Different families
Combination	Equal average	Weighted by $\alpha$	Fixed rule	Learned
Diversity from	Data resampling	Reweighting	Algorithm choice	Algorithm choice
Training	Parallel	Sequential	Parallel	Parallel, then the blender
Tuning burden	Low	High	Lowest	Moderate
Typical use	Default tabular baseline	Strongest single model	Quick easy win	Competition final step

Voting is the cheapest ensemble in this phase: if you already have three tuned models, it is one line and no new training.

pch.quizTag pch.quizDefaultTitle

1. Three classifiers predict P(class 1) of 0.45, 0.48 and 0.93. What do hard and soft voting decide?

   A Both predict class 1 B Hard predicts class 0 (two votes to one); soft predicts class 1 (mean 0.62) C Both predict class 0 D Hard predicts class 1; soft predicts class 0
   pch.quizShowAnswer

   B — Hard predicts class 0 (two votes to one); soft predicts class 1 (mean 0.62) — Hard voting counts two mild negatives against one positive. Soft voting averages to 0.62 and lets the confident member carry the weight its confidence implies.

2. Why must a stacking blender be trained on out-of-fold predictions?

   A To speed up training B Because in-sample predictions from fitted members are unrealistically good, so the blender would learn to trust members far more than it should C Because scikit-learn requires it D To handle missing values
   pch.quizShowAnswer

   B — Because in-sample predictions from fitted members are unrealistically good, so the blender would learn to trust members far more than it should — A fitted forest is near-perfect on its own training rows. The blender must see each member's performance on data that member did not train on, which is exactly what the cv parameter arranges.

3. Hard voting scored 0.8860 while its best member scored 0.9020. What went wrong?

   A A bug in VotingClassifier B Two of the four members are markedly weaker, and equal votes let them outvote the strongest C Hard voting is always worse than soft D Not enough members
   pch.quizShowAnswer

   B — Two of the four members are markedly weaker, and equal votes let them outvote the strongest — Hard voting gives every member an identical say. Members at 0.856 and 0.858 can jointly outvote one at 0.902. Soft voting recovered 1.4 of the lost points.

4. You are soft voting over logistic regression, a calibrated SVM and Gaussian naive Bayes. What should you check?

   A That all three use the same features B Naive Bayes's calibration — it routinely emits 0.9999 without meaning it, which would dominate the average C That the models are the same size D Nothing; soft voting handles it
   pch.quizShowAnswer

   B — Naive Bayes's calibration — it routinely emits 0.9999 without meaning it, which would dominate the average — Soft voting averages probabilities as if they were comparable. An overconfident member effectively gets extra votes. Wrap it in CalibratedClassifierCV or use hard voting.

🧪 Try It Yourself

Exercise 1 – Hard and soft disagree

pch.exerciseTag pch.exerciseTitle

Exercise 2 – Build a voting ensemble

pch.exerciseTag pch.exerciseTitle

Exercise 3 – Read the blender’s weights

pch.exerciseTag pch.exerciseTitle

Exercise 4 – Why out-of-fold matters

pch.exerciseTag pch.exerciseTitle

Exercise 5 – Did the ensemble actually earn its place?

pch.exerciseTag pch.exerciseTitle

Recap

• Hard voting counts labels; soft voting averages probabilities. On 0.45, 0.48 and 0.93 they give opposite answers.
• Soft voting is only as good as the members’ calibration — an overconfident member effectively gets extra votes.
• Stacking learns the combination weights instead of fixing them, and its coefficients tell you how much each member was trusted.
• The blender must be trained on out-of-fold predictions. A fitted forest scores 1.0 on its own training rows and 0.88 on unseen ones.
• Keep the blender simple; it sees only one number per member.
• Measured here: hard 0.8860, soft 0.9000, stacking 0.9000, best member 0.9020. No combination won. Always run that comparison.

Exercise 6 – Hard voting against soft voting on the same probabilities

pch.exerciseTag pch.exerciseTitle

Next

Phase 5 ends here. Continue to Phase 6 - Unsupervised Learning — the same rigour applied to problems with no labels at all, where even “is this good?” needs a new answer.