Definition

Statistical significance indicates whether the difference observed between two variants in an A/B test is likely real or due to random chance. Marketers typically require 95% confidence (p < 0.05) before declaring a winner.

Detailed Explanation

Factors affecting significance: sample size, conversion rate, and effect size (magnitude of difference). A 0.5% lift on a high-traffic page reaches significance faster than a 5% lift on a low-traffic page.

Rule of thumb: run tests until each variant has at least 100 conversions (for conversion-rate tests) or use a sample size calculator before starting.

Stopping tests early when one variant “looks ahead” inflates false positive rate — a common CRO mistake.

Nepal Context

Low-traffic Nepali business websites may need 4–8 weeks to reach significance on headline tests. Prioritize high-traffic pages (homepage, top product) or test bigger changes (layout vs. button color) to detect meaningful lifts with smaller samples.

Practical Examples

  1. Variant A: 4.2% CVR (420/10,000); Variant B: 4.8% CVR (480/10,000) — use calculator to confirm significance.
  2. Pre-calculate required sample size: baseline 3% CVR, minimum detectable effect 20% → ~15,000 visitors per variant.
  3. If not significant after 4 weeks, either extend, increase traffic, or test bolder hypothesis.

Key Takeaways

  • 95% confidence = 5% chance the result is a fluke.
  • Sample size and test duration matter more than early peeking.
  • Insignificant results are still valuable — document and move on.

Common Mistakes

  1. Stopping when first variant leads — peaking inflates false positives.
  2. Testing micro-changes on low traffic — tests never reach significance.
  3. Ignoring segment differences — mobile vs. desktop may need separate analysis.