A/B test significance
Conversions of variants A and B → z-test, p-value, verdict
Variants
Two-proportion z-test: z = (pB−pA)/√(p(1−p)(1/nA+1/nB)). p<0.05 — statistically significant difference (95%). Don't conclude on small samples: hundreds of conversions needed. Also check uplift — a difference can be significant but trivial.
Result
- A
- 5.00%
- B
- 6.08%
- Uplift
- +21.6%
- z
- 2.39
- p-value
- 0.0167
- Verdict
- Significant (95%)