Post hoc power calculator
Enter the visitors and conversions from a finished test and this returns its observed power, the number reviewers ask for after a null result. It also shows the p value that number secretly restates, and the confidence interval that answers the question people actually mean to ask. Everything runs in your browser and nothing is uploaded.
Your test result
| Arm | Visitors | Conversions | Rate |
|---|---|---|---|
| A | 1000 | 100 | 10.00% |
| B | 1000 | 120 | 12.00% |
Observed power 0.2981 is another way of writing p = 0.1529. The two numbers are locked together by the formula on this page, so the power figure adds nothing the p value did not already say. Low observed power here does not show the test was underpowered, it restates that the result was not significant. To judge the design, read the confidence interval against the smallest lift that would matter to you.
The worked example above, 100 of 1,000 against 120 of 1,000, is a 20 percent relative lift that still comes out not significant. Its observed power of 0.2981 was checked against statsmodels, which returns the same value to machine precision for the pooled formula this page uses.
The shortcut, straight from the p value
You do not need the raw counts to compute observed power, which is the first hint that something is off with the request. For a two sided z test the p value alone fixes it.
A two sided p value of 0.0500 at alpha 0.050 always converts to an observed power of 0.5000, whatever the sample size, the metric or the platform. No data beyond the p value went into that number, which is the whole problem with it.
| Two sided p value | Observed power at alpha 0.05 |
|---|---|
| 0.001 | 0.9083 |
| 0.01 | 0.7310 |
| 0.02 | 0.6430 |
| 0.03 | 0.5832 |
| 0.05 | 0.5000 |
| 0.10 | 0.3765 |
| 0.20 | 0.2493 |
| 0.30 | 0.1792 |
| 0.50 | 0.1035 |
| 0.80 | 0.0574 |
A test that lands exactly at p equal to alpha always reports an observed power of one half, and every nonsignificant result reports less. Hoenig and Heisey worked through this identity in The American Statistician in 2001, in a paper titled The Abuse of Power, and it is the reason a request to compute observed power after a null result cannot produce new information. Goodman and Berlin made the same argument for clinical trials in the Annals of Internal Medicine in 1994.
What this calculator computes
Post hoc power, also sold as observed, achieved or retrospective power, is the power the test would have had if the true difference between arms were exactly the difference you happened to observe. For a two proportion z test with observed rates p1 and p2 and arm sizes n1 and n2, the calculator evaluates
power = Φ((|p1 − p2| − zα/2·SE₀) / SE₁) + Φ((−|p1 − p2| − zα/2·SE₀) / SE₁)
where SE₀ pools the two arms under the null hypothesis and SE₁ keeps them separate under the alternative.
SE₀ = √(p̄(1 − p̄)(1/n1 + 1/n2))
SE₁ = √(p1(1 − p1)/n1 + p2(1 − p2)/n2)
The second term in the power formula is the chance of rejecting on the wrong side. It only matters at tiny effects, but keeping it makes the formula exact under the normal approximation rather than nearly exact.
The same arithmetic read in the other direction is what the p value converter above does. Observed power is a monotone function of the p value and of nothing else, which is why the two widgets on this page always agree.
How noisy observed power is, measured
The identity says observed power adds nothing beyond the p value. A separate question is how far it strays from the true power of the design, and that we measured rather than argued. We fixed a real design, a 10 percent baseline against a 12 percent treatment at alpha 0.05, set the arm size to hit each target power exactly, then simulated 10,000 complete replicate experiments per cell and computed observed power from each replicate's own data.
| n per arm | True power | 5th pct | Median | 95th pct | Within 0.10 of truth | Share significant |
|---|---|---|---|---|---|---|
| 609 | 0.20 | 0.051 | 0.200 | 0.789 | 33.9% | 19.7% |
| 1,881 | 0.50 | 0.069 | 0.500 | 0.953 | 19.5% | 49.9% |
| 3,841 | 0.80 | 0.205 | 0.799 | 0.994 | 28.7% | 79.3% |
| 6,358 | 0.95 | 0.501 | 0.950 | 0.999 | 72.6% | 95.0% |
Read the 0.80 row. The design genuinely had an 80 percent chance of detecting the true 2 point lift, yet a single run of it reports an observed power anywhere from 0.21 to 0.99 nine times out of ten, and lands within 0.10 of the truth under a third of the time. The median tracks the truth, but you only ever get one draw, and the draw is wild. A number this noisy cannot audit your design after the fact, which is the practical half of the case against it, the identity above being the theoretical half.
At true power 0.50 the spread is worst, the middle 90 percent of reported values covering nearly the whole unit interval. Observed power is least informative exactly where a test is most likely to have produced an ambiguous result, which is when people reach for it.
What to do instead of quoting observed power
The question behind a request for post hoc power is usually, did this test have a fair chance of finding an effect worth acting on. Two tools answer that and neither is observed power.
First, the confidence interval you already have. The worked example at the top spans a 0.74 point drop to a 4.74 point gain. If lifts below 2 points would not have been worth shipping anyway, the interval failing to exclude them is the finding, stated in effect units rather than laundered through a power number. A wide interval that still contains every effect you care about says the test was uninformative. A tight interval hugging zero says the effect is small, which is an answer, not a failure.
Second, design stage power at a minimally interesting effect you choose before seeing data. That is a legitimate calculation, and it is the one our power analysis calculator performs, with the required sample size falling out of it. The MDE calculator inverts it, telling you the smallest lift a sample you already have could reliably detect. For the same worked design as our simulation, 10 percent against 12 percent at alpha 0.05, the sample needed for 80 percent power is 3,841 per arm, which both that page and the table above agree on.
If the vocabulary here is new, the statistical power visualizer draws the two error regions and lets you drag the pieces, and the confidence interval calculator handles the interval arithmetic for rates and means beyond the two arm case.
How we know these numbers are right
Every figure this page displays is computed in your browser from what you type, by MIT licensed code at github.com/theluckystrike/abwex.com. Nothing is typed into the copy by hand. The static numbers above come from the committed dataset, and the calculator recomputes live values with the same formula.
The formula was cross checked against statsmodels power_proportions_2indep on three reference cases before this page was written, agreeing to machine precision, differences on the order of 1e-16. Type any row into the calculator and compare.
| Rate A | Rate B | n per arm | Observed power |
|---|---|---|---|
| 10% | 12% | 1,000 | 0.2981 |
| 10% | 12% | 200 | 0.0978 |
| 10% | 10.5% | 5,000 | 0.1307 |
The simulation study is seeded and reproducible. The generator script and its output are committed at datasets/build_posthoc_power_simulation.py and datasets/posthoc_power_simulation.json. The built in check is the last table column, the share of replicates reaching significance recovers each cell's true power to within half a point.
What has to be true for these numbers to hold
- Each visitor is counted once, converts at most once, and lands in exactly one arm. Repeat visitors or shared sessions between arms break the independence the z test assumes.
- Counts are large enough for the normal approximation, as a working rule at least ten conversions and ten non conversions in each arm. Below that, exact methods behave better.
- The test ran to its planned size. If you peeked and stopped early, the p value is already biased, and observed power inherits every problem the p value has.
- Alpha here is two sided. A one sided convention changes both the p value and the power at a given z.
- The simulation table is for the specific design stated above it. Other baselines and lifts change the numbers, though not the shape of the story.
Common questions
Read this before you act on the output
- Results are statistical estimates and not business or financial advice.
- Observed power cannot rescue or condemn a finished test. Decisions should rest on the effect estimate and its interval.
- The normal approximation degrades with tiny arms or rates near 0 or 100 percent, and the calculator says so rather than printing a number there.
- Worked examples use the inputs printed on the page and will not match your own test.
- For a decision with real stakes, have the analysis reviewed by a qualified statistician.
Related, power analysis calculator for the version of this question that has an answer, and the false discovery rate calculator for what happens to significant results when you test many metrics at once.