Interactive guide
Same slope, same start, six different paths down.
All six start at the white dot. Slide "Steps taken" to watch them move, one step at a time.
Gradient Descent Momentum Nesterov AdaGrad RMSProp Adam
—
Try the Ravine preset at a moderate learning rate. Plain gradient descent (gray) bounces wall to wall across the narrow direction, spending most of its steps fighting sideways motion instead of making progress along the valley. Momentum and Adam barely bounce at all, because they remember which direction actually kept working.
One idea, five variations on it
Every optimizer here does the same basic thing: look at the gradient, take a step against it. The six differ only in what they remember from previous steps before deciding how big this one should be.
Plain gradient descent remembers nothing, every step is sized purely off the current gradient. Everything else here is a different answer to "what's worth remembering, and how should it change the next step?"
What each one remembers
-
1
Momentum: a running velocity
Blends the new gradient into a moving average of past ones, so consistent directions build up speed and flip-flopping directions cancel out.
-
2
Nesterov: momentum that looks first
Measures the gradient at where the momentum is already about to carry it, not where it currently stands, correcting course a step earlier.
-
3
AdaGrad: a per-direction step size
Keeps a running total of how large each direction's gradients have been, and shrinks that direction's steps in proportion, permanently.
-
4
RMSProp & Adam: the same idea, but forgetful
RMSProp fixes AdaGrad's steps never growing back by averaging recent gradient sizes instead of summing forever. Adam is RMSProp plus momentum's velocity, combined.
Switch to the Saddle preset, where every path starts almost exactly on a flat plateau, the gradient there is tiny in every direction. Plain gradient descent takes a very long time to notice it should move at all, its step size is proportional to that tiny gradient. AdaGrad, RMSProp, and Adam divide by that same tiny gradient's history instead of multiplying by it, which does the opposite: it speeds them up exactly where the raw slope is weakest.
The jargons
Six algorithms, all built from the same two ingredients: a memory of past gradients, and a rule for turning that memory into a step size.
The base step size, shared by all six here so the comparison is fair. Every optimizer scales its actual step from this number.
A running exponential average of past gradients, used as the step direction instead of the current gradient alone.
A step size that's different for every parameter and changes over time, based on that parameter's own gradient history. AdaGrad, RMSProp, and Adam all do this.
A running average that weights recent values more than old ones, so it can "forget" stale history instead of accumulating it forever.
A region where the gradient is near zero without being a minimum. A common stalling point for plain gradient descent in high dimensions.
The function being minimized, drawn here as a contour plot; tight contours mean steep terrain, just like an elevation map.
This whole page is the Partial Derivatives & Gradient page's uphill arrow, walked in reverse, six different ways. And every one of these update rules collapses back to the single m ← m − η·∂E/∂m rule on the Gradient Descent page the moment you set momentum, AdaGrad, and RMSProp's memory all to zero, gradient descent is what's left when every optimizer here forgets everything.