A/B test significance
Two-proportion z-test: is the difference real
Test data
Result
p = 0.0283
Statistically significant
- A
- 5%
- B
- 6%
- Uplift
- B +20%
- z / p
- 2.19 / 0.0283
Two-sided z-test, p<0.05. Rules: don't stop a test at 'almost significant'; aim for ~1000 visits and ~50 conversions per variant minimum.