Interactive guide
Flagging fraud, without fooling yourself about how well it works
25 credit card transactions, 5 of them fraudulent. Before any model gets built, part of this data is set aside and never touched again until the very end. Cross-validation is what happens to the rest, and fraud, being rare, is exactly the case where doing it carelessly goes wrong.
Two separate questions, two separate sets of data
"Which settings work best?" gets answered using a training set, repeatedly, via cross-validation. "How well does the finished model actually work?" gets answered exactly once, using a test set that was never involved in choosing anything. Mixing the two up, even by accident, makes a model look better than it really is.
The model here is deliberately the simplest possible: a single dollar-amount threshold: flag anything above it as fraud. That's on purpose. The model isn't the point; the evaluation procedure is.
Fraud is also rare: only 5 of these 25 transactions. That scarcity is exactly what makes plain, careless shuffling dangerous, as the last section below shows directly.
The full procedure
-
1
Hold out a test set first
Set aside before anything else happens; it won't be touched again until the final step.
-
2
K-fold CV on what's left
Split the training set into K folds, rotate which one validates, average the K scores.
-
3
Retrain on the full training set
Once you're confident in the approach, train one final model on all of it, no folds this time.
-
4
Test once, on the held-out set
The only honest measure of real-world performance, because it's the only data the model never influenced.
All 25 transactions, split once, before any modeling happens.
The Test Set (right) sits outside this whole diagram from here on; it comes back only in "Final evaluation" below.
Step through each round: one fold validates (green), the rest train (blue).
—
- This round's accuracy
- —
- This round's fraud recall
- —
- Average across all 5 rounds
- —
Every training transaction, this round: which fold it's in, and what this round's threshold predicts for it.
| # | Amount | Actual | Fold | Role this round | Predicted | Result |
|---|
Final evaluation: the Test Set, touched for the first and only time.
- Final test accuracy
- —
- Final fraud recall
- —
The jargons
Every term below exists to answer one question honestly — how will this model perform on data it hasn’t seen — and each one names a specific way that honesty can quietly get compromised.
Held out before any modeling begins, and touched exactly once: the only honest read on real-world performance.
One chunk of the training set. K-fold CV rotates which fold validates while the rest train.
Of the actual fraud cases, what fraction got caught. A fold with zero fraud cases can't report this at all; it's undefined, not zero.
When one label is much rarer than another: fraud detection, disease screening, and spam are classic examples.
Builds each fold to match the whole dataset's class proportions, instead of leaving it to chance.
Letting information from validation or test data influence training, even accidentally. It makes evaluation look better than it really is.
When classes are imbalanced, plain shuffling isn't enough
Same 25 transactions, same 5 folds, but now compare how the 5 fraud cases land depending on whether the shuffle knows about class labels at all.
Regular K-fold: fraud lands wherever it happens to land
Stratified K-fold: every fold gets its fair share
fraud cases in this fold legitimate cases in this fold