Interactive guide
The direction of most spread isn't the direction of most separation
Two classes, each a noisy diagonal smear. Squash every point onto a single line and ask: can you still tell the classes apart? Below, drag that line around a full half-circle and watch the classes merge into one blur at most angles, then click into place at exactly one, where they split almost perfectly. That's what Linear Discriminant Analysis finds automatically.
Two kinds of spread, one ratio
Every projection axis has a within-class scatter (how spread out each class is along that axis: noise you want small) and a between-class scatter (how far apart the class means land along that axis: signal you want large).
LDA doesn't search by trial and error. It solves directly for the axis that maximizes Fisher's criterion: between-class scatter divided by within-class scatter, via w = Sw-1(μA − μB), where Sw is the combined within-class scatter matrix and μA, μB are the class means.
PCA finds a different axis on purpose: the one with the most total variance, computed with no knowledge of the class labels at all. When the noisiest direction in the data isn't the direction that separates the classes (exactly the setup below), PCA and LDA point in very different directions.
How LDA solves for w
-
1
Class means
μA and μB: the center of each class.
-
2
Within-class scatter Sw
How each class spreads around its own mean, pooled across both classes.
-
3
Solve w = Sw-1(μA − μB)
The mean difference, corrected for however each class is shaped.
-
4
Project and threshold
Every point becomes one number, x·w. Classify by which side of the midpoint it falls on.
The data, with the current axis
Class A Class B current axis
Separation (Fisher's criterion) vs. angle
Fisher ratio LDA optimum PCA direction
The two classes, projected onto this one axis
—
- Fisher ratio
- —
- Threshold accuracy
- —
LDA vs. PCA, on the same data
| Metric | PCA direction | LDA direction |
|---|
PCA never looks at the labels, so it has no way to know that its favorite direction (the one with the most total spread) happens to run straight through both classes' internal noise instead of between them. LDA is built from the labels from the start: it explicitly divides out each class's own scatter before asking which direction separates the means. The two methods answer genuinely different questions: "where's the variance?" versus "where's the boundary?" There's no reason to expect the same axis to answer both.
PCA vs. LDA, at a glance
They get reached for interchangeably because both turn a pile of features into fewer numbers, but they're optimizing for completely different things, and only one of them is allowed to look at the labels while doing it.
| PCA | LDA | |
|---|---|---|
| Full name | Principal Component Analysis | Linear Discriminant Analysis |
| Uses class labels? | No | Yes |
| Main goal | Preserve maximum variance | Maximize class separation |
| Type | Unsupervised | Supervised |
| Dimensionality reduction? | Yes | Yes |
| Classification? | No | Yes, via the threshold |
| Max reduced dimensions | Up to the number of features | Number of classes − 1 |
That last row is why the demo above is stuck in 1D: with only 2 classes, LDA has exactly one discriminant direction to find, no matter how many features the data started with.
The jargons
LDA is a search for one number to maximize — between-class scatter over within-class scatter — and everything below is either a piece of that ratio or a way to check that the direction it points to actually works.
How spread out each class is around its own mean, pooled across all classes: the "noise" LDA wants small along its chosen axis.
How far apart the class means are: the "signal" LDA wants large along its chosen axis.
Between-class scatter divided by within-class scatter, for a given direction: the exact quantity LDA maximizes.
The vector w that Fisher's criterion is maximized over. Projecting data onto w gives the single most class-separating 1D view of it.
Finds the direction of maximum total variance with no access to class labels: a completely different, unsupervised objective from LDA's.
How often a simple midpoint-of-the-means threshold, applied to the 1D projection, correctly classifies a point.