Interactive guide
ROC, AUC, and the threshold
By convention, a classifier calls anything scoring above 0.5 "positive." That default is only correct in one specific situation. Below, one fixed model is put to work in three settings, same predictions, same ROC curve, same AUC, and only the cost of being wrong changes. Watch what that does to the right threshold.
Four outcomes, two kinds of mistake
Every prediction at a given threshold lands in one of four boxes. Two of them are mistakes, and they fail in opposite directions.
A Type I error is a false positive: the model raised a flag on something that was actually fine. The cost lands on whoever, or whatever, got flagged by mistake: wasted effort at best, wrongful punishment of an innocent case at worst.
A Type II error is a false negative: the model stayed quiet on something it should have caught. The cost lands later, and it compounds: a real problem keeps going unaddressed until someone finds it the hard way.
Accuracy treats these as interchangeable: one wrong answer is one wrong answer. In the real world, they almost never cost the same, and mixing them up is how a "0.5 threshold" quietly becomes the wrong call.
What ROC and AUC actually show
Slide the threshold from 1 down to 0 and, at every step, plot the true positive rate against the false positive rate. That trace is the ROC curve. The AUC is the area under it: the probability the model ranks a random positive case above a random negative one. Notice what AUC doesn't include: it says nothing about which threshold to use. That's a separate decision, made below.
The jargons, in one example
Say a test screens 100 people: 20 are actually sick, 80 are healthy. It flags 22 people positive: correctly catching 18 of the 20 sick people, and wrongly flagging 4 of the 80 healthy ones. That's TP = 18, FN = 2, FP = 4, TN = 76. Every term below is computed from that same 100 people.
The cutoff score above which the model calls something positive. This whole page is about picking that number well.
Of everyone truly positive, how many did it catch?
Of everyone truly negative, how many did it wrongly flag?
Of everyone truly negative, how many did it correctly clear? The mirror image of FPR, common in medical stats.
Of everyone it flagged positive, how many actually were?
Of everyone, how many did it get right, positive or negative?
Plot TPR against FPR at every possible threshold, from 1 down to 0. Traces the full tradeoff between catching positives and raising false alarms.
Area under that curve. The probability the model ranks a random sick person above a random healthy one, independent of any single threshold.
Here is that one model: 440 labeled examples, AUC = 0.817, fixed from here on. Three price tags, three right answers.
Case 1 · The baseline
Equal-cost classifier
Positive means the model flags an item for manual review. If it flags something that didn't need it (a false positive), someone wastes a few minutes reviewing it for nothing. If it misses something that did need review (a false negative), that item just gets caught a little later, at about the same cost. Neither mistake is worse than the other here.
- Accuracy
- 0%
- Precision
- 0%
- Recall (TPR)
- 0%
- False positive rate
- 0%
y = TPR = TP / (TP+FN) x = FPR = FP / (FP+TN)
model random guess (y = x)
cost at threshold lowest-cost threshold
Here the two error types are priced the same, so the threshold that minimizes total cost lands exactly on 0.50. This is the only one of the three cases where the textbook default is actually the right answer.
Case 2 · Type I error is expensive
Criminal trial verdict
Positive means the model votes “guilty.” If it wrongly convicts an innocent person (a false positive), that harm is severe and often impossible to undo. If it wrongly lets a guilty person walk free (a false negative), that's bad too, but the justice system is deliberately designed to treat it as the lesser harm.
- Accuracy
- 0%
- Precision
- 0%
- Recall (TPR)
- 0%
- False positive rate
- 0%
y = TPR = TP / (TP+FN) x = FPR = FP / (FP+TN)
model random guess (y = x)
cost at threshold lowest-cost threshold
Because false positives are far costlier, total harm keeps falling well past 0.50, the optimum sits around 0.76. Convicting at 0.50 confidence would rack up much more damage than the model's ROC curve alone would suggest.
Case 3 · Type II error is expensive
COVID-19 screening test
Positive means the model flags the patient as infected with COVID-19. If it misses a real case (a false negative), that person walks away thinking they're clear, keeps going to work and seeing others, and unknowingly spreads the virus further. If it flags a healthy patient by mistake (a false positive), the only cost is a few days of unnecessary isolation and a confirmatory retest.
- Accuracy
- 0%
- Precision
- 0%
- Recall (TPR)
- 0%
- False positive rate
- 0%
y = TPR = TP / (TP+FN) x = FPR = FP / (FP+TN)
model random guess (y = x)
cost at threshold lowest-cost threshold
Because false negatives are far costlier, total harm keeps falling well before 0.50, the optimum sits around 0.28. Screening at 0.50 confidence would let many real infections slip through undetected.