epochs

Learn · Module 15 · Statistics and experiments · Lesson 3 of 5

A/B tests

An A/B test is the most reliable way to know if a change helped. Most mistakes happen before it starts.

4 minute read

In short
  • Random assignment means a real difference is caused by the change.
  • Decide the metric and sample size before starting; small effects need thousands of users per group.
  • Don't stop early when p dips below 0.05; report the effect size with a confidence interval.

This lesson is part of Pro

The first two lessons of every module are free. Pro opens every lesson and every project.

See Pro plansFrom ₹119 a month. No auto-renewal.