Interactive guide
Regularization: taming an overfit model
16 noisy points were sampled from one smooth curve, then fit with a degree-15 polynomial, nearly as many coefficients as there are points. Below, watch that fit go from memorizing the noise to actually tracking the curve, purely by turning up a single penalty knob.
Why more flexibility isn't free
A model with enough free parameters can always fit its training data better, including the noise. That model looks great on the data it was fit to, and unreliable on anything new. That gap is overfitting.
Regularization fights this by adding a penalty for large coefficients directly into what the model is trying to minimize. Instead of just minimizing training error, it minimizes training error + λ/α × (size of the coefficients). Turn λ/α up, and the model is charged more for chasing every last data point, so it settles for a smoother curve instead.
Three ways to measure "size"
Ridge, Lasso, and Elastic Net differ only in what "size of the coefficients" means to the penalty: squared, absolute, or a blend of both. That difference in shape turns out to matter a lot, as the sections below show.
Underfit
Too little flexibility. Misses the real pattern even in the training data.
Good fit
Tracks the underlying curve, lets small noise stay unexplained.
Overfit
Chases every point exactly, including noise that won't repeat.
The problem
One unregularized fit, 15 degrees of freedom
These 16 points were generated from a single smooth curve (shown dashed) plus a little random noise. Fit with an unregularized degree-15 polynomial (no penalty at all), the model has almost one free coefficient per data point. It bends however it needs to in order to pass close to every single one.
The result tracks the noise, not the curve, and swings far outside the range the data ever showed, right between points and at the edges. Ask this model to predict anywhere it wasn't trained, and it's likely to be badly wrong.
Coefficient size ‖β‖₂ ≈ — (versus single digits once regularized below) Error on 14 unseen test points Test MSE ≈ — (versus training error that looks almost perfect)unregularized fit true generating curve training data held-out test data
Regularization type
Ridge (L2)
Ridge adds the squared size of every coefficient to the loss. It never zeroes a coefficient out completely; it just leans on all of them proportionally, shrinking the biggest, wildest ones the hardest.
Ridge (L2) fit true curve train test
- Train MSE
- —
- Test MSE
- —
- ‖β‖₂
- —
- ‖β‖₁
- —
- Nonzero coeffs
- — / 15
loss(β) = ½×MSE + (λ/2) × Σ β²
test MSE = (1/14) × Σ(y−ŷ)², on 14 unseen points
Notice nonzero coefficients barely move; Ridge shrinks every term a little rather than dropping any of them.
β₁ … β₁₅ (standardized): bars shrink toward zero as λ/α grows
train (keeps falling) test (falls, then rises) lowest test error
Regularization type
Lasso (L1)
Lasso adds the absolute size of every coefficient instead. That corner in the penalty means it can push a coefficient to exactly zero, effectively deleting a term from the polynomial, not just shrinking it.
Lasso (L1) fit true curve train test
- Train MSE
- —
- Test MSE
- —
- ‖β‖₂
- —
- ‖β‖₁
- —
- Nonzero coeffs
- — / 15
loss(β) = ½×MSE + λ × Σ |β|
test MSE = (1/14) × Σ(y−ŷ)², on 14 unseen points
Watch nonzero coefficients as λ/α grows: Lasso can delete polynomial terms entirely, not just shrink them. See why, geometrically, this happens.
β₁ … β₁₅ (standardized): bars shrink toward zero as λ/α grows
train (keeps falling) test (falls, then rises) lowest test error
Regularization type
Elastic Net (L1 + L2)
Elastic Net splits the penalty between Ridge and Lasso. It can still zero out coefficients, but more gently than pure Lasso, and handles correlated polynomial terms more stably.
Elastic Net (L1 + L2) fit true curve train test
- Train MSE
- —
- Test MSE
- —
- ‖β‖₂
- —
- ‖β‖₁
- —
- Nonzero coeffs
- — / 15
loss(β) = ½×MSE + λ × (0.5×Σ|β| + 0.25×Σβ²)
test MSE = (1/14) × Σ(y−ŷ)², on 14 unseen points
Watch nonzero coefficients as λ/α grows: Elastic can delete polynomial terms entirely, not just shrink them. See why, geometrically, this happens.
β₁ … β₁₅ (standardized): bars shrink toward zero as λ/α grows
train (keeps falling) test (falls, then rises) lowest test error