Interactive guide
Squashing a line into a probability
Logistic regression starts exactly like linear regression: a score z = w·x + b, but never predicts that score directly. Instead it squashes z through the sigmoid function, which maps any real number onto (0, 1), and reads the result as a probability.
One feature: the sigmoid curve is the whole model
With a single feature, w and b fully determine an S-shaped curve. w controls how sharply it rises: near 0 it's almost flat everywhere, and a large |w| turns it into a near-step function. b shifts the whole curve left or right along x.
The point where the curve crosses 0.5 is the decision boundary: everything to one side gets predicted class 1, everything to the other, class 0. Fitting logistic regression means finding the w and b that make the curve pass as close as possible to 1 for every actual class-1 point and as close as possible to 0 for every actual class-0 point, measured by log loss, not by distance the way linear regression uses squared error.
48 points below, two classes along a single feature. Drag w and b yourself, or let gradient descent find them.
From a score to a probability
-
1
Compute the linear score
z = w·x + b: exactly the same score linear regression would predict directly.
-
2
Squash it with the sigmoid
σ(z) = 1 / (1 + e⁻ᶻ). Large positive z → close to 1; large negative z → close to 0; z = 0 → exactly 0.5.
-
3
Read it as P(y=1)
The output is treated as the model's confidence that this point belongs to class 1.
-
4
Threshold to classify
P ≥ 0.5 → predict class 1, otherwise class 0. (The threshold doesn't have to be 0.5; see the ROC & AUC page for what moving it trades off.)
Click anywhere to query a probability at that x.
class 1 class 0 dashed horizontal = 0.5 threshold dashed vertical = decision boundary.
Sigmoid
z = w·x + b
Plugged in
at x = 0.00, click the chart to evaluate at any point
Drag w or b, or click the chart to query a point.
- Iteration
- 0
- Log loss
- —
- Accuracy
- —
The best fit for this dataset reaches a log loss of about 0.418 (≈ 85% accuracy). Notice it never reaches log loss of exactly 0: the classes overlap, so even the best possible curve has to stay uncertain (probabilities well away from 0 or 1) right around the boundary.
Two features: the boundary is a straight line
With two features, the sigmoid itself becomes impossible to draw: it would need a third axis for probability. But its z = 0 contour (the set of points exactly at the 0.5 threshold) is still just a straight line, because z = w₁x₁ + w₂x₂ + b is linear. That's the real shape of logistic regression's decision boundary in any number of dimensions: always a straight line (or flat hyperplane), never curved.
What the sigmoid still buys you, even here, is the smooth fade in the shading below: points far from the line are shaded with high confidence, points near it stay pale, right up until the line itself, where the model is genuinely torn 50/50.
52 points, two classes. Drag either white handle to tilt the boundary, or click anywhere to query the predicted probability there.
Fitting w and b: gradient descent on log loss
-
1
Start with a guess
Any w₁, w₂, b, even ones that get the tilt backwards, as the starting line below does.
-
2
Measure how wrong every prediction is
Log loss punishes confident wrong answers far more than uncertain ones: predicting 0.99 for an actual 0 costs far more than predicting 0.6.
-
3
Step downhill on that loss
The gradient of log loss has the same clean form as linear regression's: proportional to (prediction − actual), averaged over every point.
-
4
Repeat until the line stops moving
Log loss is convex in w and b, so, just like linear regression's error bowl, there's one minimum, reachable from anywhere.
52 points, two classes. Drag either white handle, or click to query a probability.
class 1 class 0 shading = predicted probability, fading to white at 0.5.
—
Sigmoid
z = w₁·x₁ + w₂·x₂ + b
Plugged in
at (x₁, x₂) = (0.00, 0.00), click the chart to evaluate at any point
- Iteration
- 0
- Log loss
- —
- Accuracy
- —
Click the chart to query a point.
The best straight-line boundary for this dataset reaches a log loss of about 0.268 (≈ 87% accuracy). No straight line does better: the classes genuinely overlap in the middle, and no amount of retilting removes that.
The jargons
The sigmoid turns a straight line into a probability; log loss is what gets minimized to find that line in the first place; threshold and decision boundary are about turning the probability back into an actual yes-or-no answer.
σ(z) = 1 / (1 + e⁻ᶻ). Squashes any real number into (0, 1): the function that gives logistic regression its name.
The loss logistic regression minimizes: −[y·log(p) + (1−y)·log(1−p)], averaged over every point. Punishes confident wrong answers far more harshly than uncertain ones.
Where the predicted probability crosses the threshold (usually 0.5). Always linear for logistic regression: a point in 1D, a line in 2D, a flat plane beyond that.
The probability cutoff used to turn P(y=1) into an actual class prediction. Doesn't have to be 0.5; see the ROC & AUC page for how moving it trades off false positives against false negatives.
The odds of an event are p/(1−p); its logit is log of that. Logistic regression is exactly linear in the logit (log-odds = w·x + b), which is what the sigmoid inverts.
The principle behind fitting logistic regression: choose w, b that make the observed labels as probable as possible under the model. Minimizing log loss and maximizing likelihood are the same optimization.