epochs

Learn · Module 15 · Statistics and experiments · Lesson 2 of 5

Hypothesis tests

A hypothesis test asks one question: if nothing were really going on, how surprising would this result be?

3 minute read

In short
  • The p-value is how surprising your data would be if the null hypothesis were true.
  • p < 0.05 is a convention, not proof; it isn't the chance the null is true.
  • Report effect sizes and confidence intervals, not just significance.

In plain words

Start with a null hypothesis: the boring explanation, like 'this coin is fair' or 'the new button changes nothing'. Then compute the p-value: the chance of seeing a result at least as extreme as yours if the null were true. A small p-value means your result would be surprising under the boring explanation, so you doubt it.

By convention, p below 0.05 is called significant. That's a rule of thumb, not a law, and it means 'surprising if nothing's happening', not 'definitely real'.

Worked example: is the coin fair?

You flip a coin 100 times and get 60 heads.

  1. 1
    What a fair coin would do

    Expect 50 heads. The typical spread (standard deviation) is √(100 × 0.5 × 0.5) = 5.

    expected 50, spread 5
  2. 2
    How far off is 60?
    z = (60 − 50) ÷ 5 = 2.0 spreads away
  3. 3
    How likely is something that extreme, either way?

    Using the normal curve, results 2 or more spreads from the centre happen about 4.6% of the time. The exact coin-flip calculation gives 5.7%.

    p ≈ 0.046 (normal)   p = 0.057 (exact)
  4. 4
    Conclusion

    Borderline. Suspicious, but not strong evidence. 55 heads would give p = 0.32, completely ordinary.

Your turn

A test gives p = 0.03. Which statement is correct?

What to remember

Significance isn't importance: with a million users, a tiny meaningless difference can have p < 0.001. Always report the size of the effect with a confidence interval. And a non-significant result doesn't prove there's no effect; the test may simply be too small to see it.

Common mistakes

  • Saying the p-value is the chance the result is a fluke or that the null is true.
  • Treating p = 0.049 and p = 0.051 as completely different conclusions.
  • Reporting a p-value without the size of the effect.
Check yourself

Answer in your own words, out loud or on paper, before you open the model answer.

Explain a p-value to a non-technical manager.

I'd put it like this. We start by assuming the change made no difference. The p-value is how surprising our result would be if that were true. So a p-value of 0.03 means that if the change truly did nothing, we'd only see a difference this big about 3% of the time just by chance. That's unlikely enough that we usually conclude the change did something. But it doesn't tell us how big or valuable the effect is, so I'd always show the estimated improvement with a range, like 'sign-ups rose about 1.8 points, likely between 0.5 and 3'. And a high p-value doesn't prove no effect; the test may just be too small.

More to think about
  • What's the difference between statistical and practical significance?
  • What are Type I and Type II errors?

Go further

Was this lesson clear?