Interactive guide
A prediction goes forward. The blame goes backward.
Forward pass in black, each node's value. Backward pass in orange, each node's δ (blame).
positive weight negative weight δ (backward)
- Prediction ŷ
- —
- Target y
- —
- Loss = ½(ŷ−y)²
- —
Weights, layer 1 (inputs → hidden)
Weights, layer 2 (hidden → output)
—
Drag any weight and watch the change ripple forward, instantly the prediction, the loss, and every backward δ update too. Then click "Take a gradient step" repeatedly and watch the loss walk itself down to nearly zero, that's a neural network training, one step at a time, in full view.
Two passes, one chain rule
The forward pass is just arithmetic: multiply each input by its weight, add them up, squash the result through a sigmoid, and repeat for the next layer. Nothing about it is specific to learning, it's just "what does this network currently believe."
The backward pass answers a different question for every single weight: "if I nudged just this one number, how much would the loss change?" That's a derivative, and because the network is a chain of functions, each one's answer is the chain rule applied one more time. Backpropagation is just a name for computing all of those derivatives efficiently, by starting at the loss and working backward instead of computing each one from scratch.
Reading the diagram
-
1
δ is "blame," not a value
Every node's δ is ∂Loss/∂(that node's input), how much this exact spot is responsible for the final error.
-
2
The output's δ starts the chain
It's just the prediction error, scaled by the sigmoid's own slope at that point. Everything else is downstream of this one number.
-
3
A hidden neuron's δ = downstream δ × the weight it traveled through
Big weight, big share of the blame passed back. That's the entire "backward" step, repeated per connection.
-
4
A weight's own gradient = its δ × whatever fed into it
Gradient descent then just subtracts a small multiple of that from the weight, exactly the update rule from the Gradient Descent page.
Switch the training example after a few gradient steps and watch the loss jump back up, the network never saw a dataset, it only ever memorized whatever single example was in front of it. Real training repeats this exact forward-then-backward cycle over thousands of examples, averaging their gradients together; nothing about the mechanism itself changes, only how many examples get shown before each update.
The jargons
One number, computed for every node and every weight, that says which way to nudge things.
Computing the network's prediction from its inputs, layer by layer, left to right.
Computing how much each weight contributed to the error, starting at the output and working right to left.
A node's local gradient, ∂Loss/∂z, how sensitive the final loss is to that node's raw weighted sum.
A weight's own share of the blame: its downstream δ multiplied by whatever value flowed into it during the forward pass.
σ′(z) = σ(z)×(1−σ(z)): conveniently computable from the activation value alone, no need to revisit z.
w ← w − η×∂Loss/∂w for every weight at once, using exactly the gradients backpropagation just computed.