A/B Test Significance Calculator
Enter visitors and conversions for Control (A) and Variant (B). This tool uses a two-proportion z-test to estimate whether the observed lift is statistically significant.
Tip: conversions cannot be greater than visitors. Use completed sessions only for cleaner results.
What an A/B test calculator tells you
An A/B test calculator helps you evaluate whether the difference between two conversion rates is likely a real effect or just random variation. In practical terms, it answers a key question:
- Did variant B truly perform better than A?
Without statistical testing, teams often ship changes that looked good in a short window but fail to hold up over time. A calculator gives you a repeatable decision framework for product experiments, pricing tests, landing pages, email campaigns, and paid traffic funnels.
How this calculator works
This page uses a two-proportion z-test, which is a standard method for comparing two conversion rates:
- Conversion rate A = conversions A / visitors A
- Conversion rate B = conversions B / visitors B
- Difference = conversion rate B − conversion rate A
From there, the tool computes:
- Absolute lift (percentage-point change)
- Relative uplift (percent change vs. control)
- Z-score and p-value
- Confidence interval for the conversion-rate difference
How to interpret your results
1) Statistical significance
If the p-value is lower than your alpha threshold (for example, p < 0.05 at 95% confidence), your result is statistically significant. That means the observed difference is unlikely due to random chance alone.
2) Effect size
Significance is not enough. You should also care about the size of the lift. A tiny lift can be statistically significant with huge traffic but meaningless for business impact.
3) Confidence interval
The confidence interval gives you a plausible range of true effects. If the interval crosses zero, uncertainty is still high and the test may need more traffic or a longer runtime.
A practical workflow for better experiments
- Define one primary metric (e.g., checkout conversion).
- Estimate required sample size before launch.
- Run both variants simultaneously to avoid seasonality bias.
- Wait for stable sample size and test duration.
- Use this calculator to check significance and confidence interval.
- Validate against business constraints (AOV, refunds, lead quality).
Common mistakes to avoid
- Peeking too early: stopping tests as soon as one variant looks ahead.
- Uneven traffic quality: if one variant gets better traffic, results can be misleading.
- Multiple comparisons: testing many variants increases false-positive risk.
- Ignoring secondary metrics: a conversion lift that hurts retention may be a bad tradeoff.
- Tracking issues: broken analytics can invalidate conclusions.
When to trust the winner
In most teams, a reliable decision uses all three conditions:
- p-value below your threshold,
- confidence interval mostly above zero, and
- uplift large enough to matter financially.
If one of these is missing, treat the outcome as directional instead of final. In growth and product optimization, disciplined interpretation beats excitement every time.
Final note
This calculator is built for fast, practical analysis. It is excellent for binary outcomes (convert vs. not convert). For more complex cases—like revenue distributions, repeated exposure effects, or multi-armed tests—pair this with deeper statistical methods and experimentation governance.