A/B Test P-Value Calculator

Find out whether the difference between your control and variant is statistically significant. Enter the visitors and conversions for each group to compute the p-value, z-score and observed lift in real time.

Control (A)

Variant (B)

·
·
Control rate (A)·
Variant rate (B)·
Observed lift·
Z-score·
Confidence·

How the p-value is calculated

This calculator runs a two-proportion z-test, the standard significance test for A/B experiments with binary outcomes such as sign-ups, purchases or clicks. It first computes each group's conversion rate, p_A = c_A / n_A and p_B = c_B / n_B. Under the null hypothesis the two rates are equal, so the variances are estimated from a pooled conversion rate p = (c_A + c_B) / (n_A + n_B).

The standard error of the difference is SE = sqrt(p·(1−p)·(1/n_A + 1/n_B)), and the test statistic is the z-score z = (p_B − p_A) / SE. The z-score is converted to a p-value using the standard normal cumulative distribution, approximated here with the Abramowitz & Stegun error-function formula (accurate to about 7 decimal places). A two-tailed test reports 2·(1 − Φ(|z|)); a one-tailed test reports 1 − Φ(z).

The p-value is the probability of seeing a difference at least this large purely by chance if the variants truly performed the same. When the p-value falls below your chosen alpha (0.05 means 95% confidence) the result is declared statistically significant and you can reject the null hypothesis. A non-significant result does not prove the variants are equal. It usually means you need more samples. Significance ignores business impact, so always read it alongside the observed lift and absolute volume before shipping a winner.

Related Tools