Interactive guide
Two ways to be wrong, and a knob that trades between them
A supplier claims their light bulbs last 1000 hours on average (H₀, "nothing's wrong"). Suppose they're actually cutting corners and the true average is 980 hours (H₁, "there's a real defect"). Sample some bulbs, and you have to decide which world you're in, using nothing but a sample mean that wobbles from batch to batch.
The test statistic has its own distribution
A single sample mean could come from either world by bad luck. But the Central Limit Theorem says the sample mean's own spread shrinks as n grows (standard error = σ/√n), so with enough bulbs, the two worlds' sample means barely overlap at all.
Pick a critical value: a line in the sand. Sample means past it, you reject H₀. Two ways to be wrong follow immediately: Type I error (α), rejecting H₀ when it was actually true, and Type II error (β), failing to reject H₀ when H₁ was actually true.
Move the line to shrink one error and the other grows: same data, same line, opposite consequence. The only real way to shrink both is a bigger sample.
The test, in order
-
1
State H₀ and H₁
H₀: the claim ("nothing's wrong"). H₁: what you're worried is actually true.
-
2
Pick a significance level (α)
How much Type I error you're willing to accept, commonly 5%.
-
3
Collect data, compute the statistic
Here, just the sample mean of n bulbs' lifetimes.
-
4
Compare to the critical value (or check the p-value)
Past the line, or p below α, reject H₀. Otherwise, you don't have enough evidence to.
Part 1: Type I vs. Type II: drag the critical value
H₀ (true mean 1000) H₁ (true mean 980) α β
- Type I error, α (reject a fine batch)
- —
- Type II error, β (miss a defective batch)
- —
- Power, 1 − β
- —
—
Hold α at 5%: watch power climb with n
| n | 5 | 10 | 25 | 50 | 100 |
|---|---|---|---|---|---|
| α | |||||
| Power (1 − β) |
Part 2: What is a p-value?
Fixed sample of 25 bulbs. Drag the observed sample mean and watch the shaded tail ("at least this extreme, if H₀ were true"); that shaded area is the p-value.
H₀ distribution p-value (shaded tail)
- p-value
- —
- Significance level, α (reference)
- 0.05
—
A hypothesis test never proves H₀ true or false; it only asks whether the data is surprising enough, under H₀, to act as if H₁ were true instead. Every choice of critical value trades Type I error against Type II error along the exact same curve; there is no version of this test that shrinks both by moving the line. The one lever that shrinks both at once is more data, which is exactly what the power table shows happening, at a completely fixed 5% Type I rate.
The jargons
A hypothesis test is really just a bet against H₀, and every term below describes a way that bet can go right or wrong — a false alarm (Type I), a miss (Type II), or the p-value that decides whether to place it at all.
The default claim: "no effect," "nothing's wrong." A test starts by assuming it's true and asks how surprising the data would be.
What you suspect might actually be true instead: the real effect or defect a test is trying to detect.
Rejecting H₀ when it was actually true: a false alarm. Its rate is the significance level you choose ahead of time.
Failing to reject H₀ when H₁ was actually true: a miss. Shrinks as the sample size grows or the true effect gets larger.
The probability of correctly detecting a real effect when it exists: what you're actually paying for with a bigger sample.
The probability, if H₀ were true, of a sample statistic at least this extreme. Small p means the data would be a surprising coincidence under H₀.