optimizely sample size calculator

What this calculator does

This Optimizely sample size calculator helps you estimate the number of users needed to run a statistically reliable conversion test. Instead of guessing test duration, you can plan experiments based on baseline performance, expected uplift, confidence, and power.

In plain language: it answers, “How much traffic do I need before I can trust a result?”

Inputs explained

Baseline conversion rate

Your current conversion rate for the KPI in question (checkout completion, signup, click-through, etc.). If your existing page converts at 8%, baseline is 8.

Minimum detectable effect (MDE)

MDE is the smallest relative improvement worth detecting. If baseline is 10% and MDE is 10%, your detectable target is 11% (not 20%). Smaller MDE values require larger sample sizes.

Confidence level

Typically 95%. This corresponds to a Type I error rate (alpha) of 5%. Higher confidence means stricter evidence and usually larger required sample sizes.

Power

Typically 80% or 90%. Power is your chance of detecting a true effect of at least the selected MDE. Higher power reduces false negatives but increases needed traffic.

Hypothesis type

• Two-sided: detects either improvement or degradation (safest default).
• One-sided: assumes only one direction matters and needs fewer users.

How the sample size is calculated

The calculator uses a standard normal approximation for two-proportion tests:

• Build baseline proportion p1.
• Compute treatment proportion p2 = p1 × (1 + MDE).
• Use z-scores from confidence and power.
• Estimate per-variant sample size and round up.

If you run multiple variants, the tool applies a simple alpha correction so your false-positive risk is more conservative.

Why this matters for Optimizely experiments

Optimizely gives powerful experimentation workflows, but no platform can fix underpowered tests. Without enough sample size, teams often:

• Stop tests too early and ship false winners.
• Miss true improvements because traffic was too low.
• Create inconsistent learnings quarter over quarter.

Good experiment design starts with planning sample size before launch, not after peeking at interim results.

Practical tips for real-world test planning

• Pick MDE based on business value, not wishful thinking.
• Prefer fewer simultaneous variants when traffic is limited.
• Run whole-week cycles to capture weekday/weekend behavior.
• Validate event tracking before sending traffic to a test.
• Document assumptions so future teams can audit your decisions.

Common mistakes

Using total site traffic instead of eligible traffic

Only include users who can actually see and trigger the experiment’s goal.

Confusing absolute and relative lift

A 10% relative lift on a 5% baseline is 5.5%, not 15%.

Changing goals mid-test

If KPI definitions change, sample size assumptions no longer match the original design.

Bottom line

If you use Optimizely for A/B testing, a sample size calculator is one of the highest-leverage planning tools you can use. Set your assumptions clearly, estimate required traffic up front, and commit to decision rules before the test starts.