A/B testing significance calculator

A/B testing compares two versions of something — a subject line, a landing page, a survey invitation — by showing each to a random slice of your audience and measuring which performs better.

The randomisation is the whole point. Because people are assigned to A or B by chance, any systematic difference in outcome can reasonably be attributed to the variant rather than to who happened to see it.

What A/B testing can't do on its own is tell you whether a difference you observe is real. Variant B beating A by three points might be a genuine improvement, or it might be the same coin landing heads slightly more often. That's the question significance testing answers.

A result is statistically significant when it would be unlikely to occur if the two variants were genuinely identical.

The p-value quantifies that: it's the probability of seeing a difference at least as large as yours assuming there is no real difference. A p-value of 0.03 means results this extreme would appear about 3% of the time by pure chance.

By convention, p < 0.05 is treated as significant, corresponding to 95% confidence. That threshold is a convention, not a law of nature — it was popularised for convenience and it's worth remembering that p = 0.049 and p = 0.051 are essentially the same evidence.

It runs a two-proportion z-test, the standard test for comparing two conversion rates.

Step 1 — the conversion rates

Step 2 — the pooled standard error

Under the null hypothesis both variants share one true rate, so the counts are pooled to estimate it:

Step 3 — the z-score and p-value

You're testing two survey invitation emails. Version A goes to 1,000 people and gets 100 responses. Version B goes to 1,000 people and gets 130.

• Response rate A: 100 ÷ 1,000 = 10.0%
• Response rate B: 130 ÷ 1,000 = 13.0%
• Absolute difference: 3.0 percentage points
• Relative uplift: 3.0 ÷ 10.0 = +30%

Pooling gives p̄ = 230 ÷ 2,000 = 0.115, a standard error of 0.01427, and therefore z = 0.03 ÷ 0.01427 = 2.10. The two-sided p-value is 0.0355.

Since 0.0355 < 0.05, the result is significant at 95% confidence. Version B genuinely performs better.

The most common way A/B tests go wrong is stopping too early. With small samples, conversion rates swing wildly, and a variant can appear to be winning by a wide margin on nothing but noise.

Roughly, the smaller the effect you want to detect, the more traffic you need — and the relationship is quadratic. Detecting a 10% relative lift needs about four times the sample of detecting a 20% lift.

Statistical power

Power is the probability your test detects a real effect of a given size. The convention is 80%. Below that, a non-significant result tells you very little: you may have missed a genuine improvement simply because the test was too small. That's why this calculator reports power alongside the verdict.

Decide your sample size before you start, and don't peek. Checking repeatedly and stopping the moment you cross p < 0.05 dramatically inflates your false positive rate — a practice known as optional stopping.

• Stopping when it turns significant. Fix the sample size in advance and run to it. Peeking until you get the answer you want is how noise becomes a "finding".
• Testing too many things at once. Change one element per test, or you won't know what caused the difference.
• Running for an unnatural period. Behaviour differs by day of week and time of month. Run for whole weeks.
• Ignoring the practical size of the effect. With enough traffic, a 0.1-point difference becomes statistically significant and remains commercially irrelevant. Significance is not importance.
• Treating "not significant" as "no difference". Check the power. Low power means the test was inconclusive, not negative.
• Running many variants without correction. Test twenty variants at p < 0.05 and roughly one will look like a winner by chance alone.

The same test applies to plenty of research decisions beyond web pages:

• Invitation subject lines — which wording gets more responses
• Send timing — Tuesday morning versus Thursday afternoon
• Channel — email against WhatsApp or SMS
• Survey length — whether trimming five questions lifts completion
• Incentives — whether the reward actually pays for itself

One caution specific to surveys: a variant that raises your response rate can also change who responds, which shifts your results for reasons unrelated to the thing you were measuring. Check that the extra responses look like your population before adopting the winner — and confirm you have enough of them with the sample size calculator.