Cochran Armitage trend test calculator

What the tool computes

A plain chi-square test of independence ignores the order of the arms, and so does the Fisher-Freeman-Halton exact test. The trend test uses the order. It modifies the Pearson chi-squared test to incorporate a suspected ordering in the effects of the ordered groups (Wikipedia).

From your inputs the script computes the conversion rate per group and the pooled rate, the Armitage trend statistic T, its variance with N in the denominator, the Z statistic, the chi-square statistic on one degree of freedom and the two sided p value at alpha 0.05.

It also computes the same statistic with the conditional variance, which is larger, and an exact conditional p value computed by dynamic programming over all tables with the same group sizes and the same total conversions, together with the mid p value.

All of it runs in your browser. Nothing is uploaded and no request leaves the page.

How the statistic works

T is a weighted sum over the ordered groups. Wikipedia defines T as N times the t used on this page, and the standardized statistic Z is identical in the two forms (Wikipedia). Under the null hypothesis, the standardized statistic follows a large-sample standard normal distribution.

If a linear trend in the frequencies is suspected, the weights t = (0,1,2) should be used. These weights are also often used when the frequencies are suspected to change monotonically even when the trend is not necessarily linear (Wikipedia).

Compared with other implementations

When the sample size is not large, asymptotic results might not be valid, and the asymptotic p-values may differ, perhaps substantially, from the exact p-values (SAS Institute). The exact study below quantifies that on a grid of 3 to 5 groups, 10 to 100 visitors per group with equal sizes, and equally spaced scores 0 to k-1. Across the 72 grid cells, the N variance form rejected above nominal alpha 0.05 in 21 cells (worst 0.0569), the exact conditional test never did (worst 0.0465), and the mid p version in 10 cells.

SAS PROC FREQ computes exact p-values for the Cochran-Armitage test for trend for two-way tables, based on exact conditional inference for contingency tables (SAS Institute).

The two common asymptotic forms differ by a factor of N over N minus one. Royston's ptrend statistic uses N in the formula, while the score statistic uses N minus 1 (StataCorp).

R's prop.trend.test performs a chi-squared test for trend in proportions, and by default the score is chosen as the group numbers (R manual). Its signature is prop.trend.test(x, n, score = seq_along(x)), where x is the number of events, n the number of trials, and score the group score.

Exact false positive study at nominal two sided alpha 0.05

This site publishes how often each version of the test rejects when the null is true. It's computed by exact enumeration over every possible outcome with a dynamic programme over the groups, not by simulation.

Select the number of groups to see the actual rejection rate for each variant of the test, by visitors per group and true conversion rate. Every figure in the table comes from the dataset published with this page and its generator script.

Worked examples

The examples below are synthetic tables that illustrate the calculation. Each row prints its inputs so you can load it into the calculator above and reproduce the output. The p values shown were cross checked against statsmodels Table.test_ordinal_association by the published code for this page, and the same fixtures JSON is what the "load example" buttons read.

Where these numbers come from

Every number on the page is computed by the page script from the reader's inputs, or read from one of the two datasets published with the page.

The exact size study was computed by exact enumeration with a dynamic programme over the groups, not by simulation, and it was checked against brute force enumeration of every outcome before the dataset was written. The worked examples were cross checked against statsmodels Table.test_ordinal_association; that library adds 0.5 to every cell of a table containing a zero by default, and the comparison was run with shift_zeros=False to match the textbook variance used here. R wasn't available on the build machine, so the page makes no claim of an R cross check.

The two datasets and the MIT license file are published alongside the page: ca_trend_size.json (the 72-cell exact size study), ca_trend_fixtures.json (the worked-example fixtures), and LICENSE.

The calculator code is open source at github.com/theluckystrike/abwex.com, MIT licensed, and you can read and rerun every routine behind a published number.

Assumptions and what the result doesn't prove

• The trend test assumes independent observations, each visitor counted once and randomly assigned to one group.
• A non significant trend doesn't show the arms are equal, and a significant trend doesn't prove the dose causes the change.
• The trend test isn't a test of any single pair of arms.
• A p value isn't the probability that the null hypothesis is true, and it isn't the probability that a variant is better.
• When conversions are sparse, the asymptotic p value can be unreliable, and the exact conditional p value is the safer read.

The trend test has higher power than the chi-squared test when the suspected trend is correct, but the ability to detect unsuspected trends is sacrificed (Wikipedia). If you need a test that ignores order, use the chi-square test calculator or the Fisher-Freeman-Halton exact test.

Disclaimers

• Results are statistical estimates and not business or financial advice.
• A statistically significant trend is not the same as a lift that is worth shipping.
• The worked examples are synthetic tables that illustrate the calculation, not observed customer data.