Interactive guide
The gradient has to survive the whole trip backward
The Backpropagation page's network was only two layers deep, so its gradient chain was just two multiplications long. Stack far more layers and the exact same chain rule multiplies far more terms together, and multiplying many numbers that are each a little less than 1, or a little more than 1, stops looking "a little" different very fast.
One weight, one derivative, repeated L times
Make every layer a single neuron with the same shared weight w, so the whole network is one long chain. The gradient reaching the very first layer is the output layer's own gradient, multiplied by w × φ′(z) once per layer in between: δ1 = δL × ∏ (w × φ′(zi)).
If that per-layer factor is consistently below 1, the product shrinks geometrically with depth, vanishing: ten layers at 0.5 each is already a factor of roughly 0.001. If it's consistently above 1, the product grows just as fast, exploding. Sigmoid's derivative peaks at just 0.25, so a sigmoid network vanishes almost no matter what the weights are, the single biggest reason plain deep sigmoid networks were nearly impossible to train.
ReLU's derivative is exactly 1 wherever it fires, so its per-layer factor is just w itself: below 1, it vanishes; at exactly 1, the signal survives depth unchanged; above 1, it explodes, cleanly and predictably. That predictability, not immunity, is why ReLU plus careful initialization became the default.
Reading the diagram
-
1
The chain runs left (input) to right (output)
Forward pass values shown at each node, computed left to right exactly once.
-
2
The bars show |δ| flowing right to left
A log scale, because these magnitudes routinely span many orders of magnitude within one chain.
-
3
Watch the bars shrink, hold steady, or grow
All three are the same mechanism, just a per-layer multiplier under, at, or over 1.
-
4
The effective multiplier is the geometric mean
Raise it to the power of the layer count to see how the total ratio was built, one layer at a time.
Forward pass (black) left to right, then |δ| (colored) flowing back right to left.
vanishing (<1 per layer) stable (≈1 per layer) exploding (>1 per layer)
- |δ| at layer 1 (input end)
- —
- |δ| at layer L (output end)
- —
Activation function
Network depth
Shared weight w
—
—
- Ratio (layer 1 / layer L)
- —
- Effective per-layer multiplier
- —
Same L and w, all three activations
Set w = 1.0 and switch to ReLU: the bars go perfectly flat, no vanishing, no exploding, at any depth. Switch back to Sigmoid at the same w = 1.0 and the bars still collapse toward zero within a handful of layers, sigmoid's 0.25 ceiling makes vanishing close to unavoidable, no weight choice fixes it. This is exactly why ReLU networks can go far deeper than sigmoid networks ever could, and why initialization schemes (Xavier, He) exist specifically to land w near that knife-edge value of 1.
The jargons
All of these describe the same repeated multiplication, just at different scales of depth.
The backward gradient shrinks toward zero with depth, so early layers receive almost no learning signal and stop updating.
The backward gradient grows without bound with depth, producing huge, unstable weight updates.
w × φ′(z) at one layer. Whether this is above, at, or below 1 determines the entire chain's fate.
Sigmoid and tanh flatten for large |z|, driving φ′(z) toward 0 there, a second, independent source of vanishing beyond just the weight.
Weight-scaling rules chosen specifically so the per-layer multiplier starts near 1 on average, buying a deep network a fighting chance before training even begins.
A separate fix entirely: add a shortcut path around each layer so the gradient always has an alternate route with a multiplier of exactly 1.