Adjust Alpha for Multiple A/B Test Comparisons
Running several variants against one control inflates your false-positive rate. This calculator applies the Bonferroni correction so each comparison stays honest about statistical significance.
| m comparisons | Corrected threshold | Your p significant? |
|---|
How the Bonferroni correction works
When you test multiple variants at once, the chance that at least one shows a false positive grows with each comparison. With m independent tests each at alpha α, the family-wise error rate is approximately 1 - (1 - α)^m — for four variants at 0.05 that is already about 18.5%, not 5%.
The Bonferroni method fixes this by dividing your target family-wise alpha by the number of comparisons. Each individual test must now clear a stricter bar. The tool computes that corrected threshold, converts it to a per-test confidence level (1 - α/m, shown as a percentage), and compares any p-value you enter against it. It also prints the exact family-wise error rate 1 - (1 - α/m)^m after correction so you can confirm the control held near your original α.
Bonferroni is the most conservative of the multiple-comparison adjustments, which is why it trades statistical power for tight control of false positives. If you are screening dozens of metrics and can tolerate some false discoveries, a false-discovery-rate approach (Benjamini–Hochberg) recovers more power. But for a handful of A/B variants where a wrong "winner" ships real product changes, Bonferroni's simplicity and guaranteed bound make it the safe default. The sweep table below shows how the threshold tightens as you add variants, so you can see exactly how much harder each new comparison makes it to declare a winner — a planning detail most calculators omit. Use it to right-size your test count before you launch, not after.