Calculate whether your A/B test results are statistically significant. Determine confidence level, p-value, and uplift for conversion rate tests.
A/B Test Significance Calculator
Control (A)
CVR:
Variant (B)
CVR:
Uplift
Confidence
Z-Score
P-Value
Ad · 336x280
A/B Testing: Statistical Significance Explained
Statistical significance tells you whether the difference in conversion rates between your control and variant is real or just random chance. Without significance testing, you might ship a "winning" variant that was actually just lucky noise — and hurt your real conversion rate.
Key A/B Test Terms
Confidence level: Probability that the observed difference is real (not luck). 95% is the standard threshold.
P-value: Probability of seeing this result if there were no actual difference. Below 0.05 = statistically significant at 95% confidence.
Uplift: Relative improvement of Variant B over Control A in conversion rate.
Z-score: Standard deviations the observed difference is from zero. Z > 1.96 → 95% confidence.
How Much Traffic Do You Need?
Sample size depends on your baseline conversion rate, expected uplift, and confidence threshold. Rough guide: to detect a 10% uplift on a 5% conversion rate at 95% confidence, you need ~15,000 visitors per variant. Smaller uplift = more traffic needed. This is why trivial tests ("change button color") often need weeks to reach significance on low-traffic sites.
Common A/B Testing Mistakes
Stopping the test as soon as significance is reached (peeking problem — inflates false positive rate)
Running the test less than one full business week (misses weekly traffic patterns)
Testing multiple variants simultaneously without adjusting for multiple comparisons
Changing the test mid-run
Frequently Asked Questions
95% confidence (p < 0.05) is the standard for most business A/B tests. This means there's only a 5% chance the result is due to random variation. Some teams use 90% for lower-risk tests (content changes) or 99% for high-stakes changes (checkout flow, pricing). The higher your confidence threshold, the more data you need.
Run your test for at least one full week to capture daily traffic cycles, ideally 2–4 weeks. The minimum is reaching the required sample size per variant, not just hitting 95% confidence early. Stopping early (even at 95% confidence) due to 'peeking' inflates false positive rates — you may need to reach your pre-calculated sample size regardless of interim results.
Minimum Detectable Effect (MDE) is the smallest uplift you want to be able to detect reliably. Setting a realistic MDE drives your sample size calculation. A 5% conversion rate with 10% relative MDE means you want to detect a move from 5% to 5.5%. This requires ~15,000+ visits per variant at 95% confidence, 80% power.
Yes, but on different, non-overlapping parts of the funnel. Running two tests on the same page simultaneously creates interaction effects that invalidate both tests. Use multi-variate testing (MVT) if you want to test multiple elements on the same page — though MVT requires much larger traffic to reach significance.