Interactive guide
Rolling downhill, one step at a time
Fitting a line means finding the slope (m) and intercept (b) that make the error smallest. Gradient descent doesn't solve for that directly; it starts from a bad guess and repeatedly nudges m and b a little downhill, like a ball rolling toward the bottom of a bowl.
The error surface is a bowl, and it's exact
For an ordinary least-squares line, the error (mean squared error, as a function of m and b) isn't just bowl-shaped, it's a perfect paraboloid. That's a strong guarantee: no matter where you start, there is exactly one minimum, and it's downhill in every direction from anywhere else.
Gradient descent exploits that shape directly. At the current (m, b), compute the gradient: the direction the error increases fastest, and step the opposite way, scaled by a learning rate (η). Too small and it crawls; too large and it overshoots the bottom entirely, as you'll see below.
16 points, fit by hand: start at m = b = −8 (error ≈ 340) and try to reach the true optimum at m ≈ 5.31, b ≈ 0.69 (error ≈ 1.61).
How gradient descent works
-
1
Start with a guess
Any m and b, even a bad one. Gradient descent doesn't need a good starting point, just a downhill direction from wherever it is.
-
2
Compute the gradient
∂E/∂m and ∂E/∂b: how fast the error changes if you nudge each parameter. Together they point in the direction of steepest increase.
-
3
Step the opposite way
m ← m − η·∂E/∂m, and the same for b. The learning rate η controls how big a step: too large, and the step overshoots the bottom.
-
4
Repeat until it stops moving
Once the gradient is essentially zero, you're at the bottom of the bowl; further steps wouldn't change m or b.
The error surface
Exact-ellipse error contours. The ball is the current (m, b, error).
Current fit
residual (yᵢ − ŷᵢ), squared and averaged: this is the error.
A slice through the bowl
Error vs. m, holding b fixed. Fainter dots are recent steps.
Local vs. global minima: try it yourself
This bowl is unusually well-behaved because mean squared error on a linear model is always convex: one smooth valley, reachable from anywhere. Fit something more flexible, like a neural network, and the error surface can fold into many hills and valleys.
Gradient descent still just rolls downhill from wherever it starts, but now which valley it lands in depends entirely on that starting point. Click anywhere along the curve to drop a ball there and watch it roll to the nearest bottom. Try a few different spots.
Click anywhere on the curve to start.
Click along the curve: the ball starts exactly where you click and rolls downhill from there.
The jargons
One vector (the gradient) and one number (the learning rate) drive everything below — how they interact decides whether the algorithm settles into a minimum, overshoots into divergence, or gets stuck in a dip that only looks like the bottom.
The vector of partial derivatives of the error with respect to every parameter. It points in the direction the error grows fastest.
How far each step moves, as a multiple of the gradient. The single most consequential number in the whole algorithm.
The gradient shrinks toward zero and the parameters stop moving: you've reached a minimum (or something very close to one).
Steps that overshoot so far the error gets worse, not better, each iteration, usually a sign the learning rate is too high.
A single bowl with one minimum, reachable from any starting point. True for plain least-squares error, not for most deep learning loss surfaces.
A dip that's the lowest point nearby, but not the lowest possible; gradient descent can't tell the difference from inside one.