Interactive guide
The one nonlinear step between every layer
Stack as many linear layers as you want; matrix times matrix is still just one matrix. An activation function is the small nonlinear step applied after each layer that actually lets a network learn curves, not just planes. Which one you pick changes more than accuracy: it changes whether gradients survive the trip back through a deep network at all.
Why nonlinearity is the whole point
If every layer just multiplied its input by a weight matrix, a 10-layer network would collapse into one big matrix multiply, mathematically identical to a single layer. Activation functions are what break that collapse, letting depth actually buy you something.
But not every nonlinear function works equally well. During backpropagation, the local derivative of the activation function gets multiplied into the gradient at every single layer. If that derivative is tiny, the gradient shrinks every time it passes through, and multiplied across many layers, tiny numbers become essentially zero.
That one fact, what does this function's derivative look like, everywhere, explains almost the entire history of activation functions, from why Sigmoid fell out of favor to why ReLU took over to why ReLU itself needed fixing.
What to check for any activation function
-
1
Range
What values can the output take? Bounded (Sigmoid, Tanh) or unbounded (ReLU family)?
-
2
Gradient at the extremes
Does the derivative shrink toward zero far from the origin ("saturating"), or stay put?
-
3
Zero-centered?
Outputs centered on zero keep gradients balanced; all-positive outputs (Sigmoid) can bias every update in one direction.
-
4
Can the gradient hit exactly zero?
If it can and stays there, that unit stops learning entirely: permanently, for ReLU.
—
f(x)
f'(x)
—
- f(x)
- —
- f'(x): the gradient backprop sees
- —
Why deep Sigmoid/Tanh networks stall: the vanishing gradient
All three derivative curves, overlaid. Sigmoid and Tanh flatten toward zero away from the origin. ReLU's derivative is flat at exactly 1 for any positive input, no matter how large.
Sigmoid' Tanh' ReLU'
Send an input of magnitude x = 3 through a growing stack of identical layers. Each extra layer multiplies the running gradient by that layer's local derivative at that same input:
| Layers deep | Sigmoid | Tanh | ReLU |
|---|
By 10 layers, Sigmoid's gradient has shrunk by a factor of 1014, for all practical purposes, zero. Earlier layers in a deep Sigmoid network simply stop receiving any learning signal. ReLU's gradient along an active path doesn't decay with depth at all.
Dead ReLU: the same 20 pre-activations, two ways
A fixed batch of 20 neurons' pre-activation values (before the activation function is applied). Under ReLU, any negative one outputs exactly zero and has exactly zero gradient, so it can never recover, no matter what training does next.
—
Softmax: turning several scores into one probability distribution
Softmax is different from the rest: it takes a whole vector of scores (logits) and returns a vector of probabilities that sum to 1. Drag any logit below and watch every probability update together.
Sigmoid and Tanh squash everything into a bounded range and pay for it with vanishing gradients far from zero: fine for a single output layer, painful stacked ten layers deep. ReLU fixed the saturation problem on the positive side by refusing to saturate at all, but handed back a new one: any unit that lands in negative territory dies instantly and permanently. Leaky ReLU, ELU, and GELU are all answers to that same trade-off, each keeping a small but genuine gradient alive on the negative side instead of a hard zero.
| Function | Formula | Range | Zero-centered? | Gradient issue |
|---|---|---|---|---|
| Sigmoid | 1 / (1 + e−x) | (0, 1) | No | Saturates both sides: vanishing gradient |
| Tanh | tanh(x) | (−1, 1) | Yes | Saturates both sides: vanishing gradient |
| ReLU | max(0, x) | [0, ∞) | No | Exactly zero gradient for x < 0: can die permanently |
| Leaky ReLU | x if x≥0, else 0.01x | (−∞, ∞) | Roughly | Small but never-zero negative gradient |
| ELU | x if x≥0, else ex−1 | (−1, ∞) | Roughly | Smooth negative side, costs an exp() to compute |
| GELU | x · Φ(x) | ≈(−0.17, ∞) | Roughly | Smooth everywhere, most expensive to compute |
| Softmax | ezᵢ / Σⱼ ezⱼ | (0, 1), sums to 1 | — | Vector-valued: used for the output layer, not hidden layers |
The jargons
Most of what separates a good activation function from a bad one comes down to how it behaves at the extremes — whether it saturates, whether it can die, whether it’s centered on zero — not how it behaves near the middle, where the interesting cases barely differ.
An activation whose derivative shrinks toward zero as the input grows large in magnitude: Sigmoid and Tanh both saturate on both sides.
When small per-layer derivatives compound multiplicatively across many layers during backpropagation, shrinking the gradient toward zero before it reaches early layers.
A ReLU unit whose input is always negative outputs zero and has zero gradient. It can never update again, effectively removed from the network.
An activation whose output can be negative as well as positive. All-positive outputs (like Sigmoid's) can push every downstream gradient in a consistent direction.
A raw, unnormalized score fed into Softmax, before it's converted into a probability.
Adding the same constant to every logit leaves every Softmax probability unchanged; only the differences between logits matter.