How Long Should I Run My A/B Test?
At 1,000 daily visitors with a 5% conversion rate: run for 7-8 days to detect a 20% relative lift. Never stop early because it "looks significant" -- that inflates false positive rates (the peeking problem).
Duration Formula
days = required_sample_per_variant / (daily_visitors / num_variants)For a 5% baseline, 20% MDE, 95% confidence, 80% power:
- Required per variant: ~3,623 visitors
- At 1,000 daily visitors split 50/50: 500 per variant per day
- Days needed: 3,623 / 500 = 7.2 days (round up to 8)
Duration by Traffic Level
| Daily Visitors | 20% MDE | 10% MDE |
|---|---|---|
| 500 | 15 days | 59 days |
| 1,000 | 8 days | 30 days |
| 5,000 | 2 days* | 6 days |
| 10,000 | 1 day* | 3 days* |
* Even with enough samples, always run at least 7 days to account for day-of-week effects. 5% baseline, 80% power.
The Peeking Problem
Checking results repeatedly and stopping when significance appears inflates your false positive rate from 5% to as high as 30%. This happens because random fluctuations in small samples can temporarily produce "significant" results that vanish with more data. The solution:
- Calculate your sample size before starting
- Commit to running until you reach it
- Run at least one full week (7 days) regardless
- Only analyze results after the test is complete
Minimum 7-Day Rule
Even if your math says you only need 2 days of data, always run for at least 7 days. Conversion rates vary by day of week -- weekday shoppers behave differently from weekend browsers. A test that runs Monday through Wednesday may produce results that do not hold on weekends.
Use the ABWex calculator to compute exact durations for your traffic level and desired sensitivity.