Statistical Significance.

Statistical significance is a threshold that indicates test results are unlikely to be due to random chance. In A/B testing, a result is typically considered statistically significant when there is a 95% or higher probability that the observed difference between variants reflects a true underlying difference—not noise in the data. This threshold corresponds to a p-value of 0.05 or lower.

Why Statistical Significance Matters in CRO

Running A/B tests without reaching statistical significance leads to false positives—declaring a winner based on random fluctuations. A variant that appears to be winning at 70% confidence may reverse and underperform when scaled. Premature test conclusions result in implementing changes that don't actually improve performance, wasting development effort and potentially harming conversions.

Sample Size and Test Duration

The required sample size to reach significance depends on the expected effect size (the size of improvement you expect), the baseline conversion rate, and the confidence level required. Small effect sizes (1–2% improvement) require much larger samples than large effect sizes (10%+ improvement). Most A/B testing tools include sample size calculators to estimate required test duration before starting.