Interactive guide
Two rulers, measured in totally different units, made into one
Age is measured in years, income in dollars. Rescaling never mixes them together; it just takes one feature at a time and stretches or shifts its own numbers, using only that feature's own average and spread, onto a ruler that means the same thing for every feature.
One feature in, one feature out: never both at once
Below, watch age get rescaled using only age's own numbers, and income get rescaled using only income's own numbers. Neither one ever looks at the other; that's the entire point. Only afterward, once both live on the same kind of ruler, does it make sense to compare them.
Two rescalings are shown here. Standardizing centers a feature on 0, in units of "how many standard deviations from average." Normalizing squeezes a feature into the range [0, 1] instead, using its minimum and maximum. Toggle between them below.
Standardizing one feature
-
1
Find its mean and standard deviation
Computed once per feature, from that feature's whole column alone.
-
2
z = (x − mean) / std
Subtract the average, then divide by the spread. Same formula, every feature, applied on its own.
-
3
Every feature now centers on 0
A z-score of 1.0 always means "one standard deviation above average," for age or income alike.
-
4
Now the two rulers match
Only once both features speak the same units does comparing or combining them mean anything.
Standardizing
plugged in for each feature
Same idea, drawn out: two incomparable rulers, then one shared one.
Before: two different units, two different rulers
After: both on one shared z-score axis
Every person: age rescaled next to age, income rescaled next to income. The two never combine.
| Person | Age | Calculation | Age (z) | Income | Calculation | Income (z) |
|---|
The jargons
Standardization and min-max normalization are two different ways to put features on equal footing. The real question below is which algorithms actually need that footing, and which never cared about raw scale to begin with.
Rescaling a feature to (x − mean) / std, so it ends up centered at 0 with a standard deviation of 1.
A different rescaling: (x − min) / (max − min), squeezing every value into exactly [0, 1], always, for any feature.
Rescaling one feature never looks at any other feature's values; age's mean and std have nothing to do with income's.
Once every feature is on the same kind of ruler, "1 unit" means the same relative thing everywhere, which is what makes combining features (like in a distance formula) meaningful.
KNN, K-Means, SVM, PCA, and gradient descent all treat raw numeric size as meaningful; all of them need scaled features to behave sensibly.
Decision trees and their ensembles (Random Forest, boosted trees) split on thresholds per feature independently, so rescaling a feature doesn't change what they learn.