ABWex / Tools / Stratified sampling calculator

Stratified sampling calculator

Splitting a sample across strata is easy. Knowing whether the split bought you anything is the part that gets skipped. This calculator returns both allocations and the variance each one actually delivers, so the gain is a number rather than a belief.

Your strata

One stratum per line. Give a name, the size of the stratum in the population, and its standard deviation, separated by commas. Add a fourth number, the stratum mean, and the design effect becomes exact rather than conservative.

The two allocations, and when each one wins

Once you have decided how many units to sample in total, there are two standard ways to divide them between strata.

Proportional allocation gives each stratum a share of the sample equal to its share of the population.

n_h = n · N_h / N

Neyman allocation weights by size and by spread together, so a stratum that is large or noisy gets more of the sample.

n_h = n · (N_h · S_h) / sum(N_h · S_h)

Proportional is the safe default and needs no knowledge of the spreads. Neyman minimises the variance of the overall estimate for a fixed total sample, but only if your standard deviations are roughly right. If the strata all have similar spread, the two allocations converge and Neyman buys nothing. The calculator prints both, and prints the gap, so you can see which case you are in before choosing.

The number that tells you whether it was worth it

The design effect is the ratio of the variance you actually get to the variance a simple random sample of the same size would have given.

deff = Var(stratified) / Var(simple random)

Below one, stratification helped, and one divided by the design effect is the effective multiplier on your sample size. At 0.80 you are getting the precision of a simple random sample 25 percent larger. At 1.00 the strata told you nothing that random assignment would not have, and the extra operational work bought precisely nothing.

Most stratified sampling calculators do not show this number, and it is the only one that answers whether the exercise was worthwhile. Stratification pays when the thing you are measuring differs a lot between strata and relatively little inside them. If your strata are internally as varied as the population, there is no gain to find.

One detail worth stating because it trips people up. Compare proportional allocation against the within stratum variance alone and the design effect is exactly 1.000, every time, as an algebraic identity rather than a finding. The entire gain from stratifying lives in the differences between the stratum means, so a calculator that never asks for those means cannot tell you whether stratifying helped. That is why the mean is an input here, and why the output says plainly which of the two comparisons it just made.

How this applies to an experiment rather than a survey

The same arithmetic drives stratified randomisation in an online experiment, where it is usually called blocking or pre stratification. Assign users to variants within strata rather than across the whole population, and the between stratum variance stops contributing to the difference you are trying to measure.

The practical caveat is that the gain is bounded by how much of your metric's variance actually lives between strata. Splitting by device or country feels meaningful, but if conversion rate varies far more between individuals than between devices, the design effect will sit near one and the stratification is decoration. Run the numbers here before building the machinery.

For experiments the more reliable variance reduction technique is usually to use a pre period covariate instead, which is what CUPED does. The two are complementary rather than competing, and CUPED typically removes more variance because a user's own past behaviour predicts their future behaviour better than their device category does.

How we know these numbers are right

Every figure on this page is computed in your browser from the strata you enter. Nothing is typed into the copy by hand and nothing is sent to a server. The routines are MIT licensed at github.com/theluckystrike/abwex.com.

The variance of a stratified mean is the standard textbook expression, the sum over strata of the squared population weight times the stratum variance divided by its sample size. The comparison against simple random sampling uses the population variance implied by your strata, which is the within stratum variance plus the between stratum variance of the means. That decomposition is why stratification can only help when the stratum means differ.

The allocations are checked two ways in the output. Both columns are forced to sum to the total sample you asked for, and each stratum is capped at its own population size, because you cannot sample more units from a stratum than exist in it. Where that cap binds, the calculator says so instead of printing an impossible allocation.

What has to be true for these numbers to hold

Common questions

How many strata should I use?
Few enough that each still receives a usable sample. The gain from stratification comes from differences between stratum means, and adding strata that behave alike adds operational cost without reducing variance. If a stratum would receive fewer than about 30 units, fold it into a neighbour.
What if I do not know the standard deviations?
Use proportional allocation, which needs only the stratum sizes. Neyman allocation built on guessed spreads can be worse than proportional, because a wrong guess moves sample away from the stratum that actually needed it. A pilot or historical data is the usual way to get the spreads.
Why is my design effect above one?
That happens when the allocation is far from proportional and the spreads do not justify it, so sample has been moved to strata that did not need it. Check the proportional column. If proportional gives a design effect at or below one and your chosen allocation does not, the allocation is the problem rather than the stratification.
Is this the same as quota sampling?
No. Stratified sampling selects randomly within each stratum, which keeps the estimate unbiased and lets you compute a standard error. Quota sampling fills a target count by whatever units are reachable, which is not random and does not support these formulas.

Read this before you act on the output

Related, sample size calculator for the total before you split it, and CUPED variance reduction for the covariate based alternative.