A/B Test Significance Calculator
Enter visitors and conversions for the original and the variant. You get the uplift, the p-value from a two-proportion z-test and a plain verdict at your chosen confidence.
Worked examples
100 of 1,000 against 130 of 1,000
Original: 1,000 visitors, 100 conversions Variant: 1,000 visitors, 130 conversions 95% confidence, two sided
Rates 10% and 13%, uplift +30%, z-score 2.103, p-value 0.0355. Significant at 95%, but not at 99%.
The pooled rate is 11.5%, and the standard error is 0.01427.
A bigger test with a smaller gap
Original: 5,000 visitors, 200 conversions Variant: 5,000 visitors, 250 conversions
Rates 4% and 5%, uplift +25%, z-score 2.412, p-value 0.0159.
The interval for the difference is about 0.19 to 1.81 percentage points.
About A/B Test Significance Calculator
An A/B test shows two versions of a page, email or advert to different visitors and counts how many take the action you want. The two versions will almost never score exactly the same, so the real question is whether the gap is bigger than chance alone would produce. Looking at the percentages by eye is a reliable way to be fooled, especially with small numbers.
This calculator answers the question with a two-proportion z-test. It pools the two conversion rates, measures how far apart they are in standard errors (the z-score) and turns that into a p-value: the chance of seeing a gap this large if the two versions were really the same. If the p-value is below your threshold, the result is called statistically significant. It also gives the relative uplift, a confidence interval for the difference, the likely range for each rate, and a warning when the counts are too small for the method to be trusted.
How to use A/B Test Significance Calculator
- Enter the visitors and the conversions for the original version.
- Enter the same two numbers for the variant.
- Choose your confidence level, and whether you are asking if the variant is different or if it is better.
- Read the verdict, the p-value and the interval, and look at the size of the uplift before deciding.
Common uses
- Judging a landing page test
- Checking an email subject line test
- Comparing two advert versions
- Deciding whether to roll out a checkout change
Frequently asked questions
What does the p-value mean?
It is the probability of seeing a difference at least this large if the two versions really perform the same. A small p-value, below 0.05 for 95% confidence, means the gap is unlikely to be chance. It is not the probability that the variant is better.
What does confidence mean here?
The confidence level sets how strict the test is. At 95%, you accept a 5% risk of calling a difference real when there is none. Choose 99% when a wrong call would be costly.
Should I use one sided or two sided?
Two sided asks whether the versions differ in either direction and is the usual choice. One sided asks only whether the variant is better, gives a smaller p-value, and should be chosen before you see the results, not after.
Can I stop the test as soon as it is significant?
No. Checking repeatedly and stopping at the first significant result makes false wins far more likely. Work out the number of visitors needed first, and read the result once at the end.
When is the test unreliable?
The method works well when every group has at least about 5 expected conversions and 5 expected non-conversions. The page warns you when your numbers are below that.