Interactive guide
Too simple, too wild, or just right?
Every model has to choose how much it trusts the training data. Trust it too little and the model is too rigid to learn the real pattern: underfitting. Trust it too much and it memorizes the noise along with the pattern: overfitting. Below, the exact same 18 noisy points get fit by polynomials from degree 1 to degree 9. Drag the slider and watch both failure modes happen to the same curve.
Two separate ways to be wrong
Bias is a model's inability to capture the true pattern because it's too simple: a straight line trying to fit a curve will be wrong in the same systematic way no matter which 18 points it happens to see.
Variance is a model's tendency to memorize the specific noise in whichever training points it happens to get: fit a very flexible curve to one sample of 18 points, then a different sample of 18 points from the same population, and the two fitted curves barely resemble each other.
A model needs to be flexible enough to capture the real pattern (low bias) but not so flexible that it starts fitting noise (low variance). Those two goals pull in opposite directions, and that tension is the bias-variance tradeoff.
Reading the demo below
-
1
Model complexity slider
Degree 1 is a straight line; degree 9 can bend through almost anything, including noise.
-
2
Training vs. testing error
Training error keeps falling as complexity rises. Testing error draws a U: that's the tradeoff, visible directly.
-
3
The dartboard
Refit the same model on 300 fresh noisy samples and plot every prediction as a dart. Off-target = bias. Scattered = variance.
Same two ideas, three angles: if one framing doesn't click, try the next.
| Bias | Variance | |
|---|---|---|
| In plain English | Consistently wrong in the same direction, no matter what data it's shown. | Wildly inconsistent, giving a noticeably different answer nearly every time it's retrained. |
| Formally | How far the model's average prediction, across many different training sets, sits from the true value. | How much the model's prediction swings across those same different training sets. |
| Where it comes from | The model is too simple, or makes the wrong assumptions, so it can't get close even with unlimited data. | The model is too flexible, so it fits the specific random noise in whichever sample it happened to train on. |
Further down, this exact distinction gets drawn literally: off-target darts for bias, scattered darts for variance.
The fit, on this data
true function training points testing points current fit
Error vs. model complexity
Bias² Variance Total error (log scale)
Bias & variance as darts on a target
Var.
Good Fit
degree —
Underfitting
degree 1
Var.
Overfitting
degree 9
Worst of both
degree 1, n=5
Underfitting vs. Good Fit vs. Overfitting
| Metric | Underfit | Good Fit | Overfit |
|---|
The same idea shows up in classification
Illustrative only: the same 16 points, split into two classes, with a decision boundary drawn three ways.
Underfitting: a straight line can't bend around either cluster, so it misreads several points on both sides.
Good fit: one smooth curve, following the real shape of the boundary and accepting the two genuinely ambiguous points (circled) as reasonable error.
Overfitting: two extra loops reach out just to capture those same two points, at the cost of a far less trustworthy boundary everywhere else.
Push a high-bias model to fit better and its variance climbs; it starts reacting to noise it used to ignore. Pull a high-variance model back toward stability and its bias climbs; it starts missing structure it used to catch. You can't drive both to zero at once with the same model; the best you can do is find the complexity where their sum, total error, is smallest. That's the bias-variance tradeoff, and it's why "just add more flexibility" is not always the right fix.
The jargons
Bias and variance are two different ways a model can be wrong, and they move in opposite directions on the same knob — everything else here just names where on that knob a model has landed.
Error from a model being too simple to capture the true pattern: wrong in the same systematic way regardless of which training sample it sees.
Error from a model being too sensitive to the specific training sample. Fit it again on different data and the result changes a lot.
Model complexity is too low. Both training and testing error are high, because the model can't even fit the data it was trained on.
Model complexity is too high. Training error is very low (it's memorized the training set) but testing error is high: it fails to generalize.
The complexity where testing error is lowest, not where training error is lowest, which keeps falling forever as complexity rises.
Reducing bias (adding flexibility) tends to raise variance, and reducing variance (adding constraints) tends to raise bias: the same knob controls both, in opposite directions.