Free A/B Test Significance Calculator
Enter visitors and conversions for each variant. This tool runs a two-proportion z-test to estimate whether the observed conversion difference is statistically significant.
What this A/B statistical significance calculator does
This calculator helps you decide whether a conversion-rate difference between two variants (A and B) is likely real or just random noise. It is built for common growth and experimentation use cases: landing page tests, checkout changes, email subject line tests, pricing page tests, call-to-action tweaks, and other conversion optimization experiments.
Under the hood, it compares two proportions using a two-tailed two-proportion z-test. You provide traffic and conversions, and it returns:
- Conversion rates for A and B
- Absolute difference and relative lift
- Z-score and p-value
- Confidence interval for the conversion-rate difference
- A plain-language significance verdict
How the significance calculation works
1) Define hypotheses
The null hypothesis says there is no difference between true conversion rates: pA = pB. The alternative hypothesis says they are different: pA ≠ pB.
2) Compute observed conversion rates
For each variant:
- Conversion rate A = conversionsA / visitorsA
- Conversion rate B = conversionsB / visitorsB
3) Calculate pooled standard error and z-score
For hypothesis testing, the z-test uses a pooled estimate of conversion probability. Then z-score is:
z = (rateB − rateA) / standardError
A larger absolute z-score means a less likely result under the null hypothesis.
4) Convert z-score to p-value
The p-value estimates how likely it is to observe a difference this extreme (or more) if no true difference exists. If p-value is lower than alpha (for example, 0.05 at 95% confidence), the result is considered statistically significant.
How to interpret the output correctly
- Statistically significant: Evidence suggests the variants truly differ.
- Not significant: You do not yet have strong enough evidence to call a winner.
- Lift: Relative improvement of B compared with A.
- Confidence interval: Plausible range for the true difference (B − A).
If the confidence interval crosses zero, it means your data is still compatible with “no real difference.”
Practical significance vs statistical significance
A statistically significant result can still be too small to matter in business terms. For example, a +0.08% conversion lift may be significant with huge traffic but not worth engineering effort. Always combine p-value analysis with expected revenue impact, implementation cost, and user experience risk.
Common A/B testing mistakes this page helps avoid
- Stopping tests too early after seeing an initial jump
- Declaring winners with tiny sample sizes
- Confusing random fluctuation with true lift
- Ignoring practical effect size
- Running too many unplanned interim checks without adjustment
Recommended workflow for better experiments
Before launch
- Set a primary metric (for example, purchase conversion rate)
- Define minimum detectable effect and sample size target
- Choose confidence level (95% is common)
During the test
- Keep traffic allocation stable
- Avoid changing test conditions mid-flight
- Monitor data quality (tracking, bot filtering, attribution)
After completion
- Check significance and confidence interval
- Estimate expected business impact
- Validate with follow-up tests when stakes are high
Quick FAQ
Is this calculator one-tailed or two-tailed?
It uses a two-tailed test, which is the safer default for most product and marketing experiments.
Can I use it for click-through rate, signup rate, or purchase rate?
Yes. Any binary conversion metric works, as long as each user either converts or does not convert during measurement.
What if my sample size is small?
The tool will still compute results, but low sample sizes can make z-test assumptions weaker. Treat conclusions cautiously and gather more data whenever possible.