A/B Test Significance Calculator
Frequentist and Bayesian analysis, multi-variant comparison, revenue impact projections, and sample size planning. All math runs in your browser.
How the A/B Test Calculator Works
ABWex provides two distinct statistical frameworks for analyzing A/B test results. The frequentist mode uses the two-proportion Z-test, which compares conversion rates between your control and variant by computing a Z-score and corresponding p-value. The formula pools the conversion rates to estimate the common proportion under the null hypothesis, then measures how many standard deviations the observed difference falls from zero. When the p-value drops below your significance threshold (typically 0.05), you can reject the null hypothesis and declare the result statistically significant.
The Bayesian mode takes a fundamentally different approach. It models each variant's conversion rate as a Beta distribution — specifically, Beta(conversions + 1, visitors - conversions + 1) using a uniform prior. ABWex then computes P(B beats A) through numerical integration using Simpson's rule for deterministic, reproducible results. The overlapping Beta distribution curves are rendered as SVG paths so you can visually see the probability mass overlap between variants. The expected loss calculation tells you the average conversion rate you would sacrifice by choosing the wrong variant, which translates directly into business risk.
The Multi-Variant tab extends both modes to support up to four variants (A/B/C/D). It automatically generates all pairwise comparisons, highlights the winning variant, and in Bayesian mode shows the probability that each variant is the absolute best. The Sample Size pre-calculator uses the standard power analysis formula to determine how many visitors you need per variant before starting your test, and includes a duration estimator based on your daily traffic.
Features
ABWex includes frequentist Z-test with p-value, Z-score, and confidence intervals, Bayesian Beta-Binomial analysis with P(B beats A) and expected loss, multi-variant comparison for up to four variants with pairwise analysis, a revenue impact calculator that projects monthly and annual revenue changes based on your average revenue per conversion and traffic volume, a sample size pre-calculator with power curves and duration estimates, color-coded significance indicators, formula transparency showing the exact math behind every result, and full client-side processing with no server dependency.
Who Uses This
Product managers use ABWex to evaluate feature experiments and decide whether to ship changes. Growth engineers run conversion rate optimization tests on landing pages, signup flows, and pricing pages. Marketing teams compare ad creative and email subject line performance. Data analysts validate experiment results using both frequentist and Bayesian frameworks. Startups with limited traffic use the Bayesian mode to make informed decisions before reaching traditional significance thresholds. If you also need to optimize your landing page copy or compress images for faster page loads, those companion tools integrate well with the A/B testing workflow.
Privacy
All statistical calculations run locally in your browser using JavaScript. Your conversion data, visitor counts, and revenue figures are never sent to any server, never stored, and never shared. There are no cookies, no analytics trackers, and no accounts required. The source code is open on GitHub. For teams building ML-powered experimentation platforms, KappaKit provides useful development utilities.
Frequently Asked Questions
What is statistical significance in A/B testing?
Statistical significance means the observed difference between your control and variant is unlikely to be due to random chance. In A/B testing, a result is typically considered significant when the p-value is below 0.05, meaning there is less than a 5% probability the difference occurred by chance. ABWex calculates this using the two-proportion Z-test.
Bayesian vs frequentist A/B testing: which should I use?
Use frequentist testing when you have a fixed sample size and want a yes/no answer about significance. Use Bayesian testing when you want to know the probability that one variant is better, need to make decisions with smaller samples, or want to quantify expected loss from choosing wrong. Toggle between modes using the switch above the calculator.
How long should I run an A/B test?
Run your test until you reach the pre-calculated sample size. Use the Sample Size tab to determine this before starting. Stopping early because you see a significant result inflates your false positive rate. Always run for at least one full business cycle (7 days minimum) to account for day-of-week effects, and never peek repeatedly at results without adjusting for multiple comparisons.
What sample size do I need for A/B testing?
It depends on your baseline conversion rate, the minimum effect size you want to detect, and your desired power level. As a rough guide: detecting a 10% relative improvement on a 5% conversion rate at 80% power requires approximately 31,000 visitors per variant. Use the Sample Size tab for your exact numbers.
What does p-value mean in A/B testing?
The p-value is the probability of observing a difference as large as your actual result, assuming there is no real difference between variants. A p-value of 0.03 means there is a 3% chance of seeing this result if the variants perform identically. It does NOT mean there is a 97% chance the variant is better. For that probability, switch to Bayesian mode which gives you P(B beats A) directly.
Can I compare more than two variants at once?
Yes, ABWex's Multi-Variant tab supports comparing up to 4 variants (A/B/C/D) simultaneously. It compares all pairs, highlights the winner, and in Bayesian mode shows the probability that each variant is the absolute best among all variants.
How do I calculate revenue impact from an A/B test?
Enter your average revenue per conversion and monthly traffic in ABWex's Revenue Impact section. It projects monthly and annual revenue changes based on the conversion rate difference, helping you translate statistical results into business decisions.
Is ABWex free to use?
Yes, ABWex is completely free with no sign-up required. All statistical calculations run in your browser — your data is never sent to any server, ensuring complete privacy for sensitive business metrics.