False discovery rate calculator

Adjust your p values

This tool corrects p values you supply. It doesn't run the underlying test, so the test itself, its tail count and its alpha stay yours to state alongside any result. The drawn example is two sided. Adjusted values are cross checked against statsmodels and the tolerance is recorded below.

How the four corrections work

All four take the same input, your list of raw p values, and hand back adjusted values you compare against alpha. What differs is the quantity each one holds down.

Bonferroni

One multiplication, no ordering. In R's p.adjust it's the method where the p-values are multiplied by the number of comparisons, and Bonferroni, Holm, Hochberg and Hommel are all designed to give strong control of the family-wise error rate. From the other side, a Laryngoscope methods review puts it as generating a stricter significance cutoff by dividing the original significance level by the number of comparisons. statsmodels calls its bonferroni method a one-step correction, which is what it is.

Holm

Holm sorts your p values and walks up the list, so the smallest p faces the full penalty and later ranks face less. statsmodels describes its holm method as a step-down method using Bonferroni adjustments. R puts the practical conclusion this way, that there seems no reason to use the unmodified Bonferroni correction because it is dominated by Holm's method, which is also valid under arbitrary assumptions. The simulation further down agrees on power, in the one cell it covers.

Benjamini Hochberg

BH controls something different. The false discovery rate is the expected proportion of false discoveries amongst the rejected hypotheses, and that's the quantity the BH and BY methods in R's p.adjust control.

That is a weaker promise than holding the whole family, so it buys you detection. R calls it a less stringent condition than the family-wise error rate. SciPy puts the same trade in its own words, that false discovery control procedures tend to offer higher power than familywise error rate control procedures such as the Bonferroni correction.

Benjamini Yekutieli

BY is BH with a penalty for not knowing how your metrics depend on each other. It replaces the target level with that level divided by the sum of one over i across the m tests, and it controls the FDR under the most general form of dependence structure. R notes it's more conservative than BH. Reach for it when you can't argue your metrics are positively related, which is the case the assumptions section below covers.

In statsmodels the string fdr_bh selects the Benjamini/Hochberg (non-negative) procedure and fdr_by selects Benjamini/Yekutieli. The step up table in the calculator shows the mechanism instead of hiding it. Rank the p values smallest first, compare each against alpha times its rank over the number of tests, find the largest rank still under its line, and keep everything at or below that rank.

What tracking several metrics actually costs you

Track ten independent metrics with no real effect on any of them and you'll still crown a false winner 40.2% of the time. Not the 5% your alpha implies. At twenty metrics it's 64.1%.

Other calculators adjust your p values and stop. None of them tell you what watching a handful of metrics does to the odds of shipping a false winner, or what the correction costs you in detection. So I simulated it instead of quoting anyone.

The study is 36 parameter cells at 200,000 simulated experiments each, alpha 0.05, seed 20260728. It sweeps the metric count over 3, 5, 10 and 20, and the correlation between metrics over zero, 0.3 and 0.6.

The third axis is the share of metrics that are true nulls, targeted at 100%, 80% and 50%. That share is a target rather than an exact setting, because a count of true nulls has to be a whole number. At 5 metrics a 50% target is 2 of 5, which is 40%. At 3 metrics the 80% and 50% targets both round to 2 of 3, so they describe the same experiment and the picker offers it once. The cell picker below reports the count each cell actually ran, which is why it lists 33 of the 36 cells.

What those four numbers mean

Ten metrics, every one a true null, nothing corrected, and at least one comes back a winner four times in ten. That's the headline. At twenty metrics it's closer to two in three.

Correlated metrics soften the problem without fixing it. Hold the count at 20 true nulls and move from independent metrics to a correlation of 0.6, and the false winner rate drops from 64.1% to 35.5%. Better. Still nothing like 5%.

Benjamini Hochberg keeps far more of your detection than Bonferroni once the metric count grows. At 20 metrics with half carrying a real effect, BH found 66.6% of the true effects against Bonferroni's 41.3%, and its realised false discovery proportion averaged 2.5%. Holm found more than plain Bonferroni in that same cell too, 44% against 41.3%, with both holding the false winner rate under the study alpha.

Pick a cell and look at it

The known answer check

A simulation you can't check is a story. When every metric is a true null and nothing is corrected, the family-wise error rate has a closed form, 1 − (1 − α)m, where m is the number of simultaneous comparisons. That is the same quantity the Laryngoscope review of multiple hypothesis testing describes.

Measured against that formula, at 3 metrics the study got 14.3% where the formula gives 14.3%. At 5 metrics, 22.5% against 22.6%. At 10 metrics, 40.2% against 40.1%. At 20 metrics, 64.1% against 64.2%. The largest gap across the four rows is 0.14 of a percentage point, at 5 metrics. That agreement is the reason to trust the cells where no closed form exists.

What has to be true for these numbers to hold

SciPy states the guarantee carefully. When p-values correspond to independent tests, or to tests with positive regression dependencies, rejecting the hypotheses whose BH adjusted p-values fall below q controls the false discovery rate at a level no greater than q times the number of true nulls over the number tested.

Metrics that move together are usually fine, and that's a result rather than a hope. Benjamini and Yekutieli proved that BH also controls the false discovery rate when the test statistics have positive regression dependency on each of the test statistics corresponding to the true null hypotheses. Their paper says the condition covers multivariate normal test statistics with positive correlation matrix, multivariate t, and comparisons of many treatments with a single control. Revenue, orders and add to cart on the same visitors sit in that family.

Negative correlation is the gap here, and naming it beats inventing an estimate. The correlation cells in this study are positive only, so a family of strongly negatively correlated metrics is outside what was measured. For all other forms of dependency Benjamini and Yekutieli give a simple conservative modification of the procedure. That's the BY column in the calculator above, so the gap has an answer even though the simulation doesn't cover it.

Two more limits worth naming.

• The alpha argument of statsmodels multipletests is documented as the FWER, family-wise error rate. Read the label on whatever tool you use before assuming a box called alpha means what you assume it means.
• A correction cleans up multiplicity and nothing else. Not peeking, not interference between overlapping experiments, not a metric definition that drifted mid test.

How this calculator was checked

The adjusted values here were compared against statsmodels multipletests across 3,600 reject decisions, spanning families of two up to fifty simultaneous tests. Every reject decision matched. The largest absolute difference in adjusted p values was 2.22e-16, which is floating point noise rather than disagreement.

statsmodels returns a reject array that is true for hypotheses that can be rejected at the given alpha, together with p-values corrected for multiple tests, and both halves were compared. SciPy documents that its own adjusted values are comparable to those produced by the R function p.adjust and by statsmodels multipletests, so the three reference implementations agree on the values this page reproduces.

Where these numbers come from

Built by Michael Lip, who maintains the open source statistics routines behind this site, MIT licensed at github.com/theluckystrike/abwex.com. Those routines include the normal and inverse normal CDF, the Lanczos log gamma, a regularized incomplete beta by continued fraction, and Simpson rule integration for the probability that B beats A.

Nothing here is typed by hand into copy. Adjusted values come from the code in this page, recomputed from whatever you paste. The study figures are read from the dataset committed to that repository.

The run behind them, 36 cells at 200,000 experiments each on seed 20260728, was performed for this page. Both files are published so you can rerun it instead of trusting a table.

• datasets/build_fdr_simulation.py, the generator
• datasets/fdr_simulation.json, the raw output
• datasets/verify_against_statsmodels.py, the statsmodels cross check

Where a closed form exists the simulation is checked against it before publishing, which is the check printed above. The one thing this site doesn't have is a live data feed, and it never claims one. Everything you see is either your own input recomputed in front of you, or a dataset in the repository carrying its seed and its generator.

Read this before you act on the output

• Results are statistical estimates and not business or financial advice.
• Every calculation assumes visitors were randomly assigned and that each visitor counts once.
• Normal approximation results need at least five expected conversions and five expected non conversions in each variant. Below that the output is unreliable.
• A statistically significant result isn't the same as a lift worth shipping.
• A p value is the probability of seeing data this extreme if the null hypothesis is true. It isn't the probability that the variant is better.
• Worked examples use the inputs printed on the page and won't match your own test.
• These tools don't adjust for seasonality, novelty effects, overlapping experiments or interference unless the page says so.
• For a high stakes decision, have the design and the result reviewed by a qualified analyst.

Related, post hoc power calculator for the number reviewers ask for after a null result, and why the confidence interval answers it better.