Stop a test the moment it "hits significance" and you're not running one test at 5% — you're running many, and any one of them crossing p<0.05 ends the trial. This tool simulates that repeated-looks process to show the true type-I error rate and the corrected per-look threshold that pulls it back to 5%.
| Look # | Cumulative α if you stop here | Inflation vs 0.05 |
|---|
The test statistic of a running A/B experiment behaves like a Brownian motion when plotted against the information fraction t ∈ (0,1]. At each look the standardized statistic is Z(t) = B(t)/√t, where B is a standard Wiener process. Peeking means you check whether |Z| crosses the fixed critical value c = Φ⁻¹(1 − α/2) at each of the N looks; the trial stops (a "win") the first time it does. Because the increments of B are correlated across looks, the joint crossing probability has no simple product form — so this calculator runs a vectorized Monte Carlo of the Brownian bridge at the N information times, counting how often at least one look crosses. For equally spaced looks it cross-checks against the classic sequential-analysis benchmarks (Pocock 1977).
The headline number is the cumulative type-I error: the chance of declaring a false winner given the null is true. With one look it equals your nominal α. At five equally spaced looks a nominal 0.05 inflates to roughly 0.14; at ten looks it climbs past 0.19. That is the hidden tax of "just checking the dashboard."
The second output inverts the problem. Using a Pocock-style constant boundary, it binary-searches for the single per-look α' whose simulated cumulative error equals your target (default 5%). It also reports the crude Bonferroni bound α/N, which is always more conservative because it ignores the positive correlation between looks. Use the corrected threshold as a stopgap — but the honest fix is a real sequential design (O'Brien–Fleming spending, SPRT, or a Bayesian model with a proper stopping rule) rather than eyeballing a p-value.