šŸ“Š Multivariate Test Duration Estimator how long to run your MVT

Calculate required sample size & timeline for multivariate tests (MVT) with up to 10 variations.

Including control
Relative lift over baseline

šŸ“ˆ Estimated requirements

ā€”
per variation
ā€”
total sample
ā€”
estimated days
ā€”
completion date

ā³ Timeline

Start
ā€”

Adjust inputs and click calculate.

šŸ“Š Duration comparison at different MDE levels

MDE (relative)Sample per variationTotal sampleEst. daysCompletion
ā€”ā€”ā€”ā€”ā€”

ā“ Why MVTs need more traffic than A/B tests

Multivariate tests compare multiple variations simultaneously, increasing the number of comparisons. To maintain statistical power and control the false discovery rate, each variation requires a similar sample size as a single A/B test, but multiplied by the number of variations. Additionally, corrections (e.g., Bonferroni) adjust significance thresholds, further increasing required sample sizes. Less traffic per variation means longer duration.

šŸ“– Frequently asked questions

How is sample size calculated?
We use a normal approximation: n = ( (Z_Ī±/2 + Z_Ī²)Ā² * (p0*(1-p0) + p1*(1-p1)) ) / (p1-p0)Ā², with Bonferroni correction for multiple variations (Ī±' = Ī± / (variations-1)). p1 = baseline * (1 + MDE/100).
What if my daily traffic is low?
Low traffic increases duration proportionally. Consider reducing variations, increasing MDE, or lowering significance/power to get faster results.
Why does the timeline show a completion date?
Based on estimated days from today. It assumes steady traffic and no early stopping. Always plan for a fixed horizon.