Zovo Tools

What is statistical significance in A/B testing?

Statistical significance means the observed difference between your control and variant is unlikely to be due to random chance. In A/B testing, a result is typically considered significant when the p-value is below 0.05, meaning there is less than a 5% probability the difference occurred by chance. ABWex calculates this using the two-proportion Z-test: z = (p1-p2) / sqrt(p_pool*(1-p_pool)*(1/n1+1/n2)).

Bayesian vs frequentist A/B testing: which should I use?

Use frequentist testing when you have a fixed sample size and want a yes/no answer about significance. Use Bayesian testing when you want to know the probability that one variant is better, need to make decisions with smaller samples, or want to quantify expected loss. Bayesian methods give you P(B beats A) directly, while frequentist methods give you a p-value that measures evidence against the null hypothesis.

How long should I run an A/B test?

Run your test until you reach the pre-calculated sample size. Use ABWex's Sample Size Calculator to determine this before starting. For a typical 5% baseline conversion rate and 10% minimum detectable effect at 80% power, you need approximately 31,000 visitors per variant. Stopping early inflates false positive rates. Always run for at least one full business cycle (typically 7 days) to account for day-of-week effects.

What sample size do I need for A/B testing?

Sample size depends on four factors: your baseline conversion rate, the minimum effect you want to detect (MDE), desired statistical power (typically 80%), and significance level (typically 0.05). The formula is: n = (Z_alpha/2 + Z_beta)^2 * (p1(1-p1) + p2(1-p2)) / delta^2. As a rough guide: detecting a 10% relative improvement on a 5% conversion rate requires about 31,000 visitors per variant.

What does p-value mean in A/B testing?

The p-value is the probability of observing a difference as large as (or larger than) your actual result, assuming there is no real difference between variants (the null hypothesis). A p-value of 0.03 means there is a 3% chance of seeing this result if the variants perform identically. It does NOT mean there is a 97% chance the variant is better — that is a common misinterpretation. For that probability, use Bayesian analysis.

Can I compare more than two variants at once?

Yes, ABWex's Multi-Variant tab supports comparing up to 4 variants (A/B/C/D) simultaneously. It compares all pairs, highlights the winner, and in Bayesian mode shows the probability that each variant is best.

How do I calculate revenue impact from an A/B test?

Enter your average revenue per conversion and monthly traffic in ABWex's Revenue Impact section. It projects monthly and annual revenue changes based on the conversion rate difference between your control and variant, helping you make data-driven business decisions.

Is ABWex free to use?

Yes, ABWex is completely free with no sign-up required. All statistical calculations run in your browser using JavaScript — your data is never sent to any server, ensuring complete privacy for sensitive business metrics.