Interactive guide
One straight line couldn't do it. Three, combined, can.
The Perceptron page ended on a cliffhanger: XOR isn't linearly separable, so a single neuron can never solve it, no matter how long it trains. Add a hidden layer and everything changes. Below is a real, tiny network — 2 inputs, 3 hidden neurons, 1 output — training live with actual gradient descent on the exact same three datasets from that page. Watch it draw three lines at once, bend each one with a squashing function, and combine them into a boundary no single line could ever be.
Same neuron, now in layers
Every hidden neuron is still exactly the perceptron-style computation from before: multiply each input by a weight, add a bias, done. The only two things that change are (1) there are three of these running side by side instead of one, each with its own weights and its own straight line, and (2) the hard step function is replaced by a smooth tanh, so the network can be trained with calculus instead of an all-or-nothing mistake rule.
The output neuron then looks at all three hidden results and combines them — another weighted sum, squashed through a sigmoid into a 0-to-1 probability. Three straight lines, each one bent by tanh, added together: that combination can curve, which is exactly what a single line never could.
Training is backpropagation again, just run over every point in the dataset at once each step (full-batch gradient descent) instead of one dragged example at a time.
Reading the chart
-
1
Dashed lines are the hidden layer, alone
Each hidden neuron's own line: where its weighted sum crosses zero, before tanh even touches it.
-
2
The shaded background is the output, combined
Blue means the network currently predicts class 1, orange means class 0; darker means more confident.
-
3
A red ring means "wrong, right now"
That point sits on the wrong side of the current shading. Watch rings disappear as training proceeds.
-
4
Every "Step" is one full pass over all the points
Gradients are averaged over the whole dataset, then every one of the 9 weights and 4 biases moves at once.
Dashed lines: each hidden neuron alone. Shading: the trained network's combined prediction.
class 1 class 0 misclassified
- Epoch
- 0
- Accuracy
- —
- Avg. loss
- —
The network, every weight and bias labelled live
positive weight negative weight hover h1/h2/h3 to spotlight its line
—
Click through all three tabs. Separable data gets solved almost as fast as the Perceptron solved it. The overlapping dataset plateaus, the loss stops falling and a handful of points stay wrong forever, because no boundary, straight or curved, can separate points that truly overlap. XOR is the one that matters: watch the shading twist into a shape no single line could draw, until every ring disappears.
This is the entire story of "depth" in one small example. A single perceptron can only ever draw one straight line, so any dataset that needs more than one line is permanently out of reach for it, that's XOR. A hidden layer of three neurons draws three lines at once, and the output layer's job is just to combine them, which is enough to carve out a curved region. Real networks scale this idea up: more neurons per layer for more lines to combine, more layers to combine combinations of lines into combinations of curves. Same mechanism throughout, just deeper.
The jargons
Everything here is the Perceptron's vocabulary plus one addition: a hidden layer, and the smooth activation that makes it trainable.
A layer of neurons that sits between the inputs and the output, visible to neither directly, each one drawing its own line through the input space.
A smooth S-curve from −1 to 1, used here in the hidden layer so the network's output changes gradually with its weights, which gradient descent needs.
The output neuron's activation, squashing its weighted sum into a 0-to-1 probability of class 1.
The loss this network minimizes: it penalizes a confident wrong prediction far more than an unsure one, more sharply than squared error would.
Every "Step" here averages the gradient over the entire dataset before moving any weight, unlike the Perceptron's one-point-at-a-time rule.
The theorem behind why this scales: a hidden layer wide enough can approximate essentially any boundary, not just XOR's.