Learn · Module 15 · Statistics and experiments · Lesson 2 of 5
Hypothesis tests
A hypothesis test asks one question: if nothing were really going on, how surprising would this result be?
- The p-value is how surprising your data would be if the null hypothesis were true.
- p < 0.05 is a convention, not proof; it isn't the chance the null is true.
- Report effect sizes and confidence intervals, not just significance.
In plain words
Start with a null hypothesis: the boring explanation, like 'this coin is fair' or 'the new button changes nothing'. Then compute the p-value: the chance of seeing a result at least as extreme as yours if the null were true. A small p-value means your result would be surprising under the boring explanation, so you doubt it.
By convention, p below 0.05 is called significant. That's a rule of thumb, not a law, and it means 'surprising if nothing's happening', not 'definitely real'.
Worked example: is the coin fair?
You flip a coin 100 times and get 60 heads.
- 1What a fair coin would do
Expect 50 heads. The typical spread (standard deviation) is √(100 × 0.5 × 0.5) = 5.
expected 50, spread 5
- 2How far off is 60?
z = (60 − 50) ÷ 5 = 2.0 spreads away
- 3How likely is something that extreme, either way?
Using the normal curve, results 2 or more spreads from the centre happen about 4.6% of the time. The exact coin-flip calculation gives 5.7%.
p ≈ 0.046 (normal) p = 0.057 (exact)
- 4Conclusion
Borderline. Suspicious, but not strong evidence. 55 heads would give p = 0.32, completely ordinary.
A test gives p = 0.03. Which statement is correct?
What to remember
Significance isn't importance: with a million users, a tiny meaningless difference can have p < 0.001. Always report the size of the effect with a confidence interval. And a non-significant result doesn't prove there's no effect; the test may simply be too small to see it.
Common mistakes
- Saying the p-value is the chance the result is a fluke or that the null is true.
- Treating p = 0.049 and p = 0.051 as completely different conclusions.
- Reporting a p-value without the size of the effect.
Answer in your own words, out loud or on paper, before you open the model answer.
Explain a p-value to a non-technical manager.
I'd put it like this. We start by assuming the change made no difference. The p-value is how surprising our result would be if that were true. So a p-value of 0.03 means that if the change truly did nothing, we'd only see a difference this big about 3% of the time just by chance. That's unlikely enough that we usually conclude the change did something. But it doesn't tell us how big or valuable the effect is, so I'd always show the estimated improvement with a range, like 'sign-ups rose about 1.8 points, likely between 0.5 and 3'. And a high p-value doesn't prove no effect; the test may just be too small.
- What's the difference between statistical and practical significance?
- What are Type I and Type II errors?
Go further
- ArticleSeeing Theory: probability and statistics, visuallyBrown University
- PaperStatistical tests, P values, confidence intervals, and power: a guide to misinterpretationsGreenland et al., 2016