Interactive guide
Predicting from whoever's closest
K-nearest neighbors doesn't fit anything. There's no line, no bowl, no centroids to converge. It just stores every training point, and when asked about a new one, looks up whichever K points are closest and lets them answer. Classification: they vote. Regression: they average.
Classification: let the neighbors vote
Place a query point anywhere. KNN measures its distance to every single training point, takes the K closest, and asks them to vote on a class: majority wins. There's no model to inspect afterward; the "model" is just the training data itself, kept around for exactly this lookup.
K controls how much the vote gets smoothed out. At K = 1, the prediction is whatever the single nearest point happens to be, fast to react, but easily thrown off by one noisy point sitting in the wrong place. Turn K up and the boundary softens, averaging away individual outliers; turn it up too far and it starts ignoring real local structure too.
44 points, two overlapping classes below, deliberately overlapping, so K actually changes what gets predicted in the messy middle.
How KNN predicts
-
1
Store the training data
That's the entire "training" step. No parameters are fit; the data itself is the model.
-
2
Measure distance to every point
Straight-line distance from the query to each training point, every time a prediction is asked for.
-
3
Take the K closest
Sort by distance, keep the nearest K. Everything else is ignored completely, no matter how much data there is.
-
4
Combine their answers
Classification: majority vote among their labels. Regression: the average of their values. Same three steps before it, different last step.
Click anywhere to place a query point.
class A class B shaded background = what KNN predicts everywhere, at the current K.
Click the chart to place a query point.
- Vote among the K nearest
- —
- Predicted class
- —
Try K = 1 first, then drag it up past 15 or 20. At K = 1 the background is a jagged patchwork: each point carves out its own tiny territory, chasing every bit of noise. By K ≈ 15–20 (roughly half the data) the boundary smooths into something closer to a straight line down the overlap, but individual local structure gets washed out with it. There's no universally correct K; it's a real trade-off between chasing noise and ignoring signal.
Regression: let the neighbors average
Same lookup, different last step. For a query x, KNN regression finds the K training points with the closest x-values and predicts the average of their y-values, no fitted line at all, just a local average that slides along as the query moves.
At K = 1 the prediction is just whatever the single nearest point happened to record. The "curve" is a jagged staircase that touches the noise directly. Raise K and it smooths into something closer to the underlying trend, at the cost of flattening real curvature the data actually has.
26 points sampled from a noisy curve below. Click anywhere along it to query a specific x, or just watch the background curve reshape itself as K changes.
The one difference from classification
-
1
Same lookup
Distance to every point, keep the K closest: identical to classification. Here, "closest" just means closest in x.
-
2
Average instead of vote
ŷ = mean of the K neighbors' y-values. A number instead of a class: the only real change.
-
3
Repeat at every x to draw the curve
Slide the query across the whole range and connect the predictions: that traces the full KNN regression curve.
Click along the chart to query an x.
training points the K nearest to your query shaded band = their x-range dashed lines carry each neighbor's y to the query bold green line = their average, which becomes ŷ.
Click anywhere on the chart to query an x.
- Query x
- —
- Predicted y (mean of K nearest)
- —
The jargons
K is the only knob KNN has, and most of the terms below are about what happens as you turn it — from a jagged, memorized boundary at K=1 toward a smoother, blurrier one as K grows, until the curse of dimensionality makes “nearest” stop meaning much at all.
How many neighbors get consulted. The only real "setting" in the whole algorithm; everything else is just distance and arithmetic.
KNN does no work up front: "training" is just storing the data. All the computation happens at prediction time, per query.
How "closest" gets defined: straight-line (Euclidean) distance here, though other metrics exist and change what counts as a neighbor.
The line separating where KNN would predict one class versus another. For KNN it's not smooth or simple; it's exactly as jagged as the data and K make it.
Small K: low bias, high variance. It reacts to every point, including noise. Large K: high bias, low variance, smoother, but blurs real local patterns too.
"Nearest" stops meaning much in very high dimensions: with enough features, every point ends up roughly equidistant from every other. KNN degrades badly there.