Interactive guide
Why a committee of weak models beats one strong guess
In 1906, at a country fair in Plymouth, 800 people guessed the weight of an ox. No individual guess was particularly good, but the median of all 800 guesses came within 1% of the animal's true weight. The statistician who happened to record this, Francis Galton, had stumbled on the entire idea behind ensemble learning: a crowd of mediocre, independent guesses, combined the right way, reliably beats any one of its members.
The four panels below are four different answers to "combined the right way, how, exactly?" They go in order, starting with the simplest one first.
Each step trades a little simplicity for a smarter way of combining models, and the four panels below follow this exact order.
Simple combination: vote, or just average
Before any of the three ideas below, there's an even simpler one: train a few genuinely different algorithms on the exact same data, and combine their outputs with a fixed, un-learned rule. For classification that rule is a vote; for regression, an average.
Hard voting counts each model's final class guess and takes the majority. Soft voting averages their predicted probabilities first, and only then decides a class, which keeps "I'm 51% sure" and "I'm 99% sure" from counting the same the way hard voting forces them to.
Three different models below: a straight line, K-nearest neighbors, and a small decision tree.
Simple combination, step by step
-
1
Train different algorithms, same data
No resampling, no reweighting: every model just sees the same training set and does its best.
-
2
Classification: vote
Hard voting: majority of their class guesses. Soft voting: average their probabilities, then threshold.
-
3
Regression: average
Simple average weights every model equally, whether or not that's actually fair to how good each one is.
-
4
The catch: the rule is fixed, not learned
If one model is much worse, a plain vote or average still listens to it exactly as much as the others; nothing here down-weights it automatically. (Stacking, at the end, fixes exactly this.)
Classification: click a point to see each model's vote and the two combined results.
class 1 class 0 linear KNN tree soft vote
Click the chart to see each model's vote at that point.
- Linear · KNN · Tree accuracy
- —
- Hard voting accuracy
- —
- Soft voting accuracy
- —
Every point, every model's vote, and both combined results. Green = matches the true class, red = doesn't.
Regression: drag the sliders to weight each model, or reset to a plain average.
a data point linear KNN tree weighted average
Weights are normalized to sum to 1 automatically.
- Linear · KNN · Tree error (MSE)
- —
- Weighted average error (MSE)
- —
Every point, each model's prediction, and the current weighted average.
Notice the plain average (equal weights) doesn't beat the best individual model here; it just splits the difference. Try dragging more weight onto whichever model has the lowest error and watch the combined error drop below all three individually. That's the exact gap stacking (at the end of this page) closes automatically, by solving for the best weights instead of guessing them.
Bagging: average away the noise
A single decision tree, grown deep enough to fit real structure, also fits the noise. Resample the training data slightly and you'll grow a noticeably different tree. That instability is variance, and it's the exact thing bagging targets.
Bagging (bootstrap aggregating) grows many trees, each on its own bootstrap sample: a random resample of the training data, drawn with replacement, so some points appear twice and others not at all. Each tree overfits its own sample in its own way. Average enough of them together and those idiosyncratic errors cancel out, leaving the real signal.
Grow a few trees yourself and watch the average settle down, first on a classification task (vote share becomes a predicted probability), then on the same regression staircase as before.
How bagging works
-
1
Draw a bootstrap sample
Same size as the original dataset, sampled with replacement, so it's a slightly different dataset every time.
-
2
Grow a full tree on it
No pruning back: let it overfit. That's expected; it's what makes each tree different from the others.
-
3
Repeat, many times over
Each tree sees a different resample, so each makes different mistakes.
-
4
Average every tree's prediction
(Or vote, for classification.) A random forest is exactly this, plus one extra trick: each split only considers a random subset of features too.
The sample → tree branch repeats N times, independently; no tree ever sees another tree's output.
Classification: grow a few trees; the vote share becomes a predicted probability.
class 1 class 0 true class (hidden from the trees)
one bootstrap tree's vote average vote so far
Grow a tree to begin.
- Trees grown
- 0
- Latest tree's accuracy
- —
- Voting accuracy of all trees
- —
Every row of the training data, and how many times the latest bootstrap sample picked it (0, 1, 2, or more, sampled with replacement).
Regression: grow a few trees, then watch the bold average line settle onto the true staircase.
a data point true staircase (hidden from the trees)
one bootstrap tree average of all trees so far
Grow a tree to begin.
- Trees grown
- 0
- Latest tree's error vs. true function
- —
- Average of all trees' error
- —
Every row of the training data, and how many times the latest bootstrap sample picked it (0, 1, 2, or more, sampled with replacement).
Every individual tree is a rough, blocky guess: right in places, badly wrong in others, and wrong differently each time, in both panels above. None of that matters once you average (or vote) over enough of them: the errors that point in different directions cancel, and what's left is much closer to the truth than any single tree, without ever changing what a single tree is capable of learning.
Boosting: correct the mistakes, one weak learner at a time
Bagging builds many strong-ish trees independently and averages them. Boosting does the opposite: it builds many weak learners (here, a "stump," a tree with just one split) one after another, where each new stump's entire job is to fix what the ensemble so far still gets wrong.
Concretely: start by predicting the overall mean for every point. Compute the residuals: how wrong that is, point by point. Fit a stump to those residuals, shrink it by a learning rate, and add it to the running prediction. New residuals, new stump, repeat. This is gradient boosting: the same "step downhill" idea as the Gradient Descent page, just taking its steps in the space of functions instead of two numbers.
Add a few rounds yourself, first on the same classification task bagging just used, then on the regression staircase.
How boosting works
-
1
Start dumb
The first prediction is just the mean of every y: no information used at all yet.
-
2
Compute the residuals
actual − predicted, for every point. This is what the ensemble is still getting wrong.
-
3
Fit a weak learner to the residuals
A single-split stump, here, just complex enough to explain a little of what's left over.
-
4
Shrink it, add it, repeat
Scale the new stump by a learning rate before adding it in: smaller steps, more of them, generalize better than a few large ones.
↻ Repeat: the updated prediction's residuals feed the next round's stump.
Classification: add weak learners one at a time, or run them all at once.
class 1 class 0 true class cumulative prediction
Ready: add the first weak learner.
- Rounds
- 0
- Training accuracy
- —
Every point's current prediction and residual: the next round's stump is fit to this residual column.
Regression: add weak learners one at a time, or run them all at once.
a data point true staircase cumulative prediction so far
current residual (what the next stump will target)
Ready: add the first weak learner.
- Rounds
- 0
- Training error (MSE)
- —
Every point's current prediction and residual: the next round's stump is fit to this residual column.
Notice the training error can only go down, round after round: each new stump is fit specifically to whatever's still wrong, so it can always help at least a little. That's also boosting's risk: run it long enough on any dataset and it will eventually start fitting noise the same way an unbagged deep tree does. In practice, the number of rounds is a tuned hyperparameter, stopped early on purpose.
Stacking: learn how to combine different models
Bagging and boosting both combine many copies of the same kind of weak model. Stacking instead trains several genuinely different base models (below, a straight line and a K-nearest-neighbors curve) and then learns a meta-model that combines their predictions, instead of just averaging them blindly.
The simplest possible meta-model is exactly what you built on the linear regression page: fit a line (here just one weight, since there are only two inputs) through the base models' predictions to best match the real answer. That's stacking with a linear meta-learner, and it's real: it's the same normal-equations calculation, just with model outputs standing in for features.
Drag the blend yourself, or let it solve for the best one, on the classification task first, then the regression staircase.
How stacking works
-
1
Train several different base models
Different enough that they make different mistakes: a smooth global model and a flexible local one, say.
-
2
Collect their predictions as new features
Every training point now has two "features": what the line predicts, and what KNN predicts.
-
3
Fit a meta-model on those predictions
Here, a single blend weight found by least squares, the same closed-form solve as ordinary linear regression.
-
4
The meta-model can down-weight a weak base model
If one base model is much worse here, the fitted blend will lean away from it automatically; nobody has to tell it to.
Classification: drag the slider to blend the two base models, or find the best blend automatically.
class 1 class 0 linear KNN (K=3) blend
Drag the slider, or find the best blend.
- Linear model accuracy
- —
- KNN model accuracy
- —
- Blended accuracy
- —
The meta-model's actual inputs: each base model's prediction per point, and the resulting blend.
Regression: drag the slider to blend the two base models, or find the best blend automatically.
a data point base model: linear base model: KNN (K=3) blend
Drag the slider, or find the best blend.
- Linear model error (MSE)
- —
- KNN model error (MSE)
- —
- Blended error (MSE)
- —
The meta-model's actual inputs: each base model's prediction per point, and the resulting blend.
The straight line can't bend to follow sharp regions or steps at all, so on its own it's the weakest model in both panels above. Watch what the optimal blend does about that: it doesn't split the weight evenly; it leans heavily toward KNN, automatically discovering that the linear model has very little to contribute to this data, without being told so directly.
The jargons
Bagging and boosting solve the same problem — combine many models — two different ways: bagging averages away variance across resampled data, boosting chains weak learners onto each other’s mistakes, and the rest of the terms below belong to one camp or the other.
A model built by combining several other models' predictions, rather than relying on any single one.
A resample of the training data, drawn with replacement, the same size as the original: the "bootstrap" in bagging.
How much a model's predictions would change if trained on a different sample of the same data. Bagging specifically targets this.
A model barely better than guessing on its own, a single-split stump here. Boosting's whole point is that many weak learners, added up correctly, can be strong.
Actual value minus current prediction. Boosting's next weak learner is trained to predict exactly this.
How much of each new weak learner actually gets added to the ensemble. Smaller values need more rounds but generalize better: the same trade-off as gradient descent's step size.
Stacking's two layers: base models make the original predictions; the meta-model learns how to best combine those predictions into one.
Bagged decision trees, plus one addition: each split considers only a random subset of features, decorrelating the trees further.
The boosting method demonstrated above, generalized: for squared error, the "residual" a new learner targets is exactly the negative gradient of the loss.