Interactive guide
Two ordinary slopes, held at a right angle.
Drag the point. The arrow always points uphill.
contour lines (equal height) your current height gradient (uphill)
∂f/∂x: slice at fixed y
∂f/∂y: slice at fixed x
—
- f(x, y)
- —
- ∂f/∂x
- —
- ∂f/∂y
- —
- |∇f|
- —
∇f = (0.00, 0.00)
The gradient isn't a third, separate idea, it's just the two side slopes packaged into one vector: ∇f = (∂f/∂x, ∂f/∂y). Drag the point along a single contour line and watch the arrow keep pointing perpendicular to it, contour lines are always the "flat" directions, so uphill has to be at a right angle to them.
Freeze one variable, differentiate the other
∂f/∂x is nothing exotic, it's the ordinary derivative of a one-variable function, the one you get by nailing y down at its current value and only letting x move. The left slice chart is that one-variable function; its tangent line's slope is ∂f/∂x, full stop. Same story for ∂f/∂y, with the roles swapped.
The gradient vector's direction is the single direction that increases f the fastest, its magnitude is exactly how fast that fastest increase is. Every other direction from the same point increases f more slowly, or decreases it, than the gradient direction does.
Reading the contour plot
-
1
Tight lines mean steep terrain
Contour lines close together mean height changes fast over a short distance, exactly like elevation lines on a map.
-
2
The arrow is always perpendicular to the local contour
Moving along a contour changes nothing about height, by definition, so the fastest-change direction has to be crosswise to it.
-
3
Longer arrow, steeper terrain
The arrow's length tracks |∇f|; near a peak, valley, or saddle point it shrinks toward zero.
-
4
On the saddle, uphill in x can be downhill in y
The two side slices can have opposite-sign slopes at once, that's exactly what makes it a saddle instead of a hill or a bowl.
Every side slice here is the exact same tangent-line construction from the Derivatives & Integration page, just sliced out of a surface instead of drawn from a single curve. And this is the arrow that Gradient Descent spends its entire time walking backward along, downhill is nothing but "the opposite of the arrow drawn here."
The jargons
Two slopes, held at a right angle, and the map contour lines they come from.
The ordinary derivative of f with every other variable held fixed, the slope of the one-variable slice through the current point.
The vector of all the partial derivatives, (∂f/∂x, ∂f/∂y) here. Points in the direction f increases fastest.
A curve where f is exactly constant, every point on it has the same height, like one ring on an elevation map.
The gradient direction, by construction: no other direction from the same point increases f as quickly.
Anywhere ∇f = 0, both partial derivatives vanish at once. Peaks, valleys, and saddle points are all critical points.
A critical point that's a minimum along one direction and a maximum along another, the "ripples" and "saddle" presets both have these.