Interactive guide
Bayes' theorem, run once per word and multiplied together
This is the same posterior-from-prior-times-likelihood idea as the Bayes' theorem page, except now there are several pieces of evidence (a message's words) instead of just one (a test result). The only new idea is "naive": treat each word as independent of the others, given the class, so their evidence can just be multiplied.
Why "naive"
In real text, words are not independent: "free" and "money" show up together more than chance would predict. Naive Bayes ignores that on purpose: it assumes that once you know the message is spam, each word's presence is an independent coin flip. That assumption is false, and the classifier works well anyway, because it usually only needs to get the ranking right (which class is more likely), not the exact probability.
Everything below comes from just counting words in 20 labeled training messages (8 spam, 12 ham), with a small correction (Laplace smoothing) so a word that never appeared in one class doesn't force its probability to a hard zero.
This model uses a Bernoulli version: a word's absence counts as evidence too. A message with none of the usual spam words isn't neutral; it's active evidence for ham, exactly like a clean medical test result is evidence of health, not just "no information."
Classifying one message
-
1
Start from the prior
P(spam) and P(ham), straight from the training split, before reading a single word.
-
2
Multiply in every word's evidence
For each vocabulary word, present or absent, multiply by however likely that state is under each class.
-
3
Work in log space
Multiplying many small probabilities underflows fast; add their logs instead, exactly as real implementations do.
-
4
Normalize and compare
Turn the two class scores back into a probability that sums to 100%, then predict whichever is higher.
What the model learned from training data: how spam-indicative each word is.
P(word | spam) P(word | ham)
- Training messages
- —
- Prior P(spam)
- —
- Prior P(ham)
- —
- Accuracy on its own training data
- —
Words like "free" and "money" show up mostly in spam, so their spam-bar is long. Words like "meeting" and "project" barely show up in spam at all; their ham-bar dominates instead.
Each word's current pull toward spam (right, red) or ham (left, blue), including words you leave off.
Bars sum exactly to the total log-odds below; this chart is the formula, drawn out term by term.
Log-odds (why we add logs instead of multiplying)
- P(spam | this message)
- —
- Predicted label
- —
Try the "free urgent meeting" preset: "free" and "urgent" push toward spam, "meeting" pushes toward ham, and the three nearly cancel out: the model lands almost exactly on a coin flip. That's not a bug. When the evidence is genuinely mixed, an honest classifier should say so instead of forcing a confident answer.
The jargons
“Naive” names one simplifying assumption — that words are independent given the class — and everything else below is either how that assumption gets applied to text (bag of words) or a patch for its rough edges (Laplace smoothing).
A classifier built from Bayes' theorem plus one extra assumption: every feature is independent of the others, given the class.
Representing a message as which vocabulary words it contains, ignoring order entirely. "Free money" and "money free" look identical.
Adding 1 to every count before dividing, so a word that never appeared in a class's training data gets a small nonzero probability instead of exactly zero.
log(P(spam)) − log(P(ham)), built by summing per-word terms. Positive favors spam, negative favors ham, zero is a dead-even split.
A Naive Bayes variant that scores whether each vocabulary word is present or absent, as opposed to a Multinomial model, which only cares about words that actually appear.
The "naive" part: treating word presence as uncorrelated given the class. Almost never exactly true, but often close enough for the classifier to still rank classes correctly.