Bayesian vs Frequentist A/B Testing: When to Use Which

By Michael Lip · April 2025 · 9 min read

The frequentist approach gives you a p-value. The Bayesian approach gives you a probability. These sound similar but lead to different decisions in practice. We ran the same three datasets through both modes on ABWex and found cases where the two methods lead to opposite conclusions.

How the Two Methods Work

Frequentist (Z-Test)

The frequentist approach asks: "If there were no real difference between A and B, how likely would I be to see a result this extreme?" That probability is the p-value. If the p-value is below your significance threshold (typically 0.05), you reject the null hypothesis and declare the result statistically significant.

The formula ABWex uses:

z = (p_B - p_A) / sqrt(p_pool * (1 - p_pool) * (1/n_A + 1/n_B))
p-value = 2 * (1 - normalCDF(|z|))

Bayesian (Beta-Binomial)

The Bayesian approach asks: "Given the data I observed, what is the probability that B is actually better than A?" It models each variant's conversion rate as a Beta distribution and computes P(B beats A) directly via numerical integration.

Control: Beta(conversions_A + 1, visitors_A - conversions_A + 1)
Variant: Beta(conversions_B + 1, visitors_B - conversions_B + 1)
P(B > A) = integral of P(B_rate > A_rate) over both distributions

Scenario 1: Large Sample, Clear Winner

Data: Control: 50,000 visitors, 2,000 conversions (4.00%). Variant: 50,000 visitors, 2,200 conversions (4.40%).

Frequentist result: z = 2.99, p-value = 0.0028. Statistically significant. 95% CI for the difference: [0.14%, 0.66%]. Relative improvement: +10.0%.

Bayesian result: P(B beats A) = 99.8%. Expected loss if choosing A: 0.38%. Beta distributions are well-separated with minimal overlap.

Verdict: Both methods agree strongly. When you have large samples and a real effect, the two approaches converge. This is the easy case.

Scenario 2: Small Sample, Marginal Difference

Data: Control: 500 visitors, 20 conversions (4.00%). Variant: 500 visitors, 28 conversions (5.60%).

Frequentist result: z = 1.20, p-value = 0.2302. Not significant. You cannot reject the null hypothesis. The standard recommendation is to keep testing.

Bayesian result: P(B beats A) = 88.5%. Expected loss if choosing B: 0.22%.

Verdict: Here the methods diverge. The frequentist test says "not enough evidence." The Bayesian test says "88.5% chance B is better with very low expected loss." If you need to make a business decision now and the cost of choosing wrong is low, the Bayesian answer is more actionable. If you need rigorous evidence (e.g., for a scientific paper), the frequentist answer is correct: you do not have enough data.

Need timestamp analysis for your experiment data? Try EpochPilot's time & date tools.

Scenario 3: Large Sample, Tiny Effect

Data: Control: 100,000 visitors, 3,000 conversions (3.00%). Variant: 100,000 visitors, 3,080 conversions (3.08%).

Frequentist result: z = 1.32, p-value = 0.1864. Not significant at alpha = 0.05.

Bayesian result: P(B beats A) = 90.6%. Expected loss if choosing A: 0.07%.

Verdict: Another divergence. With 200,000 total visitors, you have high statistical power. The tiny 0.08 percentage point difference is probably real (90.6% Bayesian probability), but it is so small that the frequentist test correctly identifies it as not meeting the significance bar. The question becomes: is a 0.08% improvement worth implementing? The Bayesian expected loss (0.07%) helps you answer that in business terms.

When to Use Which

Use Frequentist When:

You can commit to a fixed sample size before starting (use the Sample Size calculator)
You need a binary "significant or not" answer for reporting
Regulatory or scientific contexts require traditional hypothesis testing
You want to control your false positive rate precisely
You are running many tests and need to manage family-wise error rate

Use Bayesian When:

You need to make decisions before reaching ideal sample sizes
You want to know the probability that B is better, not just whether the difference is significant
You care about expected loss (the cost of being wrong)
You want to peek at results during the test without inflating false positive rates
Your stakeholders find "90% probability B is better" more intuitive than "p = 0.03"

The Practical Bottom Line

For most product teams, start with Bayesian for day-to-day decisions and use frequentist for high-stakes tests where you need to control error rates. The two methods answer different questions. The frequentist asks "is there enough evidence?" The Bayesian asks "what should I believe?" Both are valid questions, and the right one depends on your context.

Try both modes on your own data using ABWex's calculator. Toggle between frequentist and Bayesian to see how the conclusions differ for your specific numbers. For more details, see Evan Miller A/B test calculator. For more details, see VWO Bayesian testing guide.

For validating model experiments in ML pipelines, see EpochPilot.