Most wrong conclusions don't come from bad maths. They come from asking too many questions or looking at the data in the wrong slices.
3 minute read
In short
Testing many things guarantees some false wins: 20 tests at 5% gives a 64% chance of at least one.
Simpson's paradox: a different mix of groups can reverse the overall result.
Correlation suggests ideas; only randomised tests show cause.
Trap 1: testing many things
Each test has a 5% chance of a false alarm. Test 20 different metrics or user segments, and the chance at least one looks significant by pure luck is 1 − 0.95²⁰ = 64%. Somebody will find a 'win' in almost any experiment if they look in enough places.
Fixes: choose the main metric in advance, treat surprising wins in secondary metrics as ideas to re-test, and use corrections like Bonferroni (divide 0.05 by the number of tests) when you must test many.
Trap 2: Simpson's paradox
Two checkout designs, A and B, measured on phones and computers.
segment
A
B
Phone
20 of 200 = 10%
110 of 1,000 = 11%
Computer
300 of 1,000 = 30%
64 of 200 = 32%
Overall
320 of 1,200 = 26.7%
174 of 1,200 = 14.5%
1
B wins in every segment
Better on phones and better on computers.
2
A wins overall, by a lot
How?
3
The mix differs
A's users were mostly on computers, which convert far better anyway. B's were mostly on phones. The overall number compares the mix of devices, not the designs.
4
The fix
Randomise properly so both groups have the same mix, and compare within segments when the mix differs.
Trap 3: correlation isn't cause
Users who use the search feature spend more. Does search make them spend more, or do keen shoppers use more features? Without a randomised test, you can't tell. Observational data suggests ideas; experiments confirm them.
Your turn
An analyst tests a change across 20 user segments and reports that it significantly helped users in Kerala aged 25 to 34 (p = 0.04). What should you conclude?
What to remember
Decide what you're testing before you look. Be suspicious of results found by slicing. Check whether groups differ in mix. And separate 'this is associated with that' from 'this causes that'.
Common mistakes
Slicing results by many segments after the test and reporting whichever looks significant.
Applying no correction when many metrics are tested at once.
Treating a surprising secondary result as a confirmed finding instead of an idea to re-test.
Check yourself
Answer in your own words, out loud or on paper, before you open the model answer.
What is the multiple comparisons problem?
Every significance test has a chance of a false positive, usually 5%. If you run many tests, those chances add up. With 20 independent tests on something that has no real effect, the chance that at least one comes out significant is 1 minus 0.95 to the 20th, about 64%. So if you check 20 metrics or 20 user segments, you'll very likely find a 'winner' by luck alone. To handle it, I'd pick one primary metric before the test, treat other metrics as supporting evidence, use a correction like Bonferroni, dividing 0.05 by the number of tests, when many comparisons really matter, and re-test surprising findings in a new experiment.
More to think about
Explain Simpson's paradox with an example.
Users who use feature X retain better. Should we push everyone to use X?