Interactive guide
The one line that minimizes total squared error
Unlike the Gradient Descent page's iterative search, ordinary least
squares (the "linear regression" you'd get from
sklearn.LinearRegression) doesn't need to roll downhill
at all; the best slope and intercept come out of two sums, in one
shot. This page is about what "best" means, and how much a single
point can bend it.
Least squares: minimizing the sum of squared residuals
For any candidate line, each point has a residual: the vertical gap between what actually happened (yᵢ) and what the line predicts (ŷᵢ). Square every residual and add them up: that total is the SSE, and it's exactly what linear regression minimizes. Squaring does two things at once: it makes big misses count far more than small ones, and it turns the problem into one solvable directly, without searching.
R² rescales that same idea into something easier to read: the fraction of the data's total variation the line explains, from 0 (no better than just guessing the average every time) to 1 (passes through every point exactly).
22 points below. Drag either white handle to tilt your own line, then compare it to the exact answer.
How OLS solves it directly
-
1
Write the error as a function of m, b
SSE(m, b) = Σ(yᵢ − (m·xᵢ+b))², a smooth bowl, exactly like the one on the Gradient Descent page.
-
2
Set both partial derivatives to zero
At the bottom of that bowl, ∂SSE/∂m = 0 and ∂SSE/∂b = 0 simultaneously: two linear equations in m and b.
-
3
Solve them: the normal equations
m = Σ(xᵢ−x̄)(yᵢ−ȳ) / Σ(xᵢ−x̄)², then b = ȳ − m·x̄. No guessing, no iterating.
-
4
That's the only minimum there is
SSE is convex in m and b, so this closed-form point is the global minimum; gradient descent would eventually reach the exact same place, just slower.
Drag either white handle to tilt the line.
a squared residual: side length = |error|, area = its contribution to SSE exact best fit (appears after you snap to it)
Drag either handle to change the line.
- Slope, intercept (m, b)
- —
- Sum of squared errors (SSE)
- —
- R² (variance explained)
- —
The starting line simply predicts the average y for every x; that's the baseline R² is measured against, so it always scores exactly R² = 0.
The exact least-squares line is ŷ = 2.19x + 4.53, reaching R² = 0.72. No other straight line scores lower total squared error than this one, not because it was searched for, but because it's the algebraic solution.
Leverage: one point can outweigh all the others
Least squares treats every point's squared error as equally important, but that doesn't mean every point has equal influence on where the line ends up. A point far from the center of x acts like the end of a long lever: a small nudge there swings the line far more than the same nudge would near the middle.
The highlighted point below starts sitting right on the trend the other 13 points define. Drag it anywhere , straight up, straight down, or out to either edge, and watch the line refit itself live, in real time, to accommodate it.
Why position matters, not just distance
-
1
The fit always passes through (x̄, ȳ)
Every least-squares line passes through the mean point exactly, so the line pivots around that spot as any single point moves.
-
2
Distance from x̄ determines leverage
A point near the middle of x barely changes the pivot's angle. A point far to either side has a long lever arm: the same vertical drag swings the slope much further.
-
3
Try it at the center, then at an edge
Drag the point near x̄ up and down: barely anything happens. Drag it to the far left or right edge and repeat: the whole line tilts.
Drag the highlighted point anywhere; the line refits live.
fixed points drag this one
—
- Slope, intercept (m, b)
- —
- R² (variance explained)
- —
- Leverage of the dragged point
- —
The jargons
SSE is what the line minimizes and R² is how to grade the result — but neither one can tell you what leverage and influential point are for: whether a single point is quietly doing all the work.
The gap between an actual value and the line's prediction: yᵢ − ŷᵢ. Positive if the point sits above the line, negative if below.
Every residual, squared, then added up. The exact quantity ordinary least squares minimizes.
1 − SSE/SST, where SST is the SSE of the flat mean-baseline. Reads as "the fraction of the data's spread this line explains": 0 to 1 for any reasonable fit.
The closed-form solution for m and b that makes both partial derivatives of SSE zero simultaneously: what "solving" linear regression actually means, no iteration involved.
How much a point's position (in x alone, before even looking at its y) lets it pull the fitted line. Points far from x̄ have high leverage regardless of whether they turn out to be outliers.
A point with both high leverage and a y-value that breaks the trend: the combination that actually bends the fit. High leverage alone, on an on-trend point, changes very little.