Interactive guide
Finding the widest street between two classes
A Support Vector Machine doesn't just draw a line between two classes; it finds the one line with the most empty space on either side of it. Read how it does that, then try to beat it by eye in the demo below.
Classification: pick the line with the biggest margin
Plenty of lines can separate two groups of points perfectly. SVM argues that not all of them are equally good: a line that barely squeezes between the two groups is fragile: a new point, only slightly different from the ones it learned on, can land on the wrong side. A line with lots of empty space around it is far more likely to keep classifying new points correctly.
That empty space is the margin: the width of the widest strip you could draw around the line without touching a single point. SVM's whole objective is to maximize it. The points that end up exactly on the edge of that strip, holding it in place, are the support vectors: remove any other point and the boundary wouldn't move at all; remove a support vector and it would.
How SVM actually finds it
-
1
Consider every line that separates the classes
Any line with no points on the wrong side is a legal candidate; there are infinitely many.
-
2
Measure each candidate's margin
Find the single closest point to the line on either side. That distance, doubled, is the margin.
-
3
Solve for the widest one
Formally: minimize ‖w‖² subject to yᵢ(w·xᵢ+b) ≥ 1 for every point. A smaller ‖w‖ means a wider margin.
-
4
Read off the support vectors
At the optimum, only the closest point(s) on each side satisfy that constraint exactly: those are the support vectors.
-
5
If nothing separates cleanly, allow a few exceptions
Add slack variables and a penalty C: this is the "soft margin," covered in the glossary below.
24 points, two classes. Drag either white handle to tilt the line.
class A class B misclassified nearest point (defines d(A)/d(B))
SV = actual support vector, shown once you reveal the optimal line
—
- Distance to nearest class B point
- d(B) —
- Distance to nearest class A point
- d(A) —
- Current margin width
- margin = d(A) + d(B) —
—
The widest possible margin for this dataset is 1.8311: that's what an SVM actually solves for: not just a separating line, but the one line that maximizes this number.
Regression: the same idea, turned into a tube
SVM isn't only for drawing boundaries between classes. Support Vector Regression (SVR) fits a continuous line the usual way, but changes what counts as "error." Instead of penalizing every point that isn't exactly on the line, SVR wraps an ε-insensitive tube of half-width epsilon (ε) around the fitted line: any point that lands inside the tube is treated as a perfect prediction and costs nothing at all.
Concretely: each dot below is one student, hours studied on the x-axis, the exam score they actually got on the y-axis. Widen ε and two things happen together: the shaded tube grows, and the blue line itself gets flatter, since a wider tube lets SVR settle for a less steeply-tilted fit while still keeping everyone inside it. Any student whose real score already falls inside the tube is treated as a correct-enough prediction and ignored completely.
Only the students sticking out of the tube matter, and, just like the classification case, those are exactly the points that get to call themselves support vectors, labeled SV on the chart. Everyone comfortably inside the tube could move around freely without changing the fit one bit; only the ones poking through are actually holding the line in place.
How SVR actually fits it
-
1
Pick a tolerance, ε
Decide how far a prediction can be from the truth before it counts as an error at all.
-
2
Fit the flattest line that stays inside the tube
Formally: minimize ‖w‖² subject to |yᵢ − (w·xᵢ+b)| ≤ ε for every point: the flattest fit that satisfies this, not the one that chases every point.
-
3
Allow a few points to poke through
Slack variables let some points sit outside the tube, penalized by how far out they are, times a cost C.
-
4
Read off the support vectors
Only points on the tube's edge or outside it affect the fit; everything comfortably inside contributes nothing.
22 students. Drag ε below and watch two things at once: the tube resizes, and the line itself refits to the flattest one that still fits inside it.
inside the tube (ignored) SV = support vector (outside the tube)
- Current fit
- —
- Points inside the tube
- —
- Support vectors (outside the tube)
- —
Widen ε and watch the slope in "Current fit" shrink: with more slack to work with, SVR settles for a flatter line instead of tilting hard to hug every point. Only the students still poking through the tube (the support vectors) have any say in where that line ends up; everyone comfortably inside could move around freely without changing the fit at all.
When no line will do: the kernel trick
Everything above assumed a straight line could separate the two classes. Plenty of real data isn't shaped like that: the classic example is one class forming a ring around the other. No straight line, at any angle or position, separates a ring from the dot in its middle.
The kernel trick's answer: stop trying in 2D. Lift every point into 3D by adding a third coordinate, z = x²+y², literally how far each point sits from the center. The inner ring stays low; the outer ring, being farther out, rises higher. Once lifted, a single flat, horizontal plane separates them perfectly. Drag the slider below to watch that happen.
How the kernel trick works
-
1
Notice no line works
The classes aren't linearly separable in the original space, no matter how the line is drawn.
-
2
Choose a feature map, φ
A function that adds dimensions. Here, φ(x,y) = (x, y, x²+y²) adds one: distance from the center.
-
3
Separate with a plane in the new space
In 3D, the two classes now sit at different heights, so an ordinary flat plane separates them.
-
4
The "trick": never actually compute φ
SVM's math only ever needs dot products φ(a)·φ(b). A kernel function computes that value directly from a and b in the original space: the higher-dimensional vectors are never built.
24 points, two rings. Drag the slider to lift them into 3D.
outer ring (class A) inner ring (class B)
—
At lift = 0 you're looking straight down: the two rings overlap in every direction, so no line through this view separates them. Drag the slider up and the view tilts while every point rises by its own x²+y²: the outer ring, being farther from the center, lifts higher than the inner one, and a flat plane can finally sit between them.
The jargons
Everything here traces back to one idea, maximize the margin, not just accuracy, and the rest of the vocabulary covers what happens when the data won’t cooperate (soft margin), can’t be split by a straight line at all (kernel trick), or the task is regression instead of classification (ε-tube).
The decision boundary itself: a line in 2D, a flat plane in 3D, and a "hyperplane" in anything higher-dimensional. Same idea throughout, whatever splits the space into two sides.
The width of empty space around the boundary before you hit a point. SVM maximizes this, not just accuracy on the training data.
A point that sits exactly on the edge of the margin. Only these points determine where the boundary goes; every other point could be deleted with no effect.
Requires every single point correctly separated, no exceptions. Only works when the classes are actually linearly separable, as in the demo above.
Real data usually isn't perfectly separable. A soft margin allows some points inside the margin (or even on the wrong side) for a cost, controlled by a parameter C. Small C tolerates more violations for a wider margin; large C punishes every violation, hugging the data more tightly.
When no straight line can separate the classes, a kernel implicitly projects the points into a higher-dimensional space where one can, without ever computing that projection directly. See the ring-lifting demo above.
The regression version of the margin: a band of width 2ε around the fitted line where errors are simply ignored. Only points outside it are penalized.