What Is Statistical Significance?
Statistical significance tells you whether an observed difference between two proportions is bigger than chance alone would readily produce. The calculator runs a two-proportion z-test using the pooled proportion and reports the p-value and the statistical confidence (1 − p).
It also gives the absolute difference, the relative uplift of B over A, a confidence interval for the difference, Cohen's h as an effect size and the observed power at this effect. Significance alone does not say whether a difference is large enough to matter, so read the interval and effect size too.
Below you will find the formulas, a variables table, a worked example and answers to common questions such as what p-value to require and why stopping a test early distorts results.
How to Use the Statistical Significance Calculator
- Enter visitors and conversions for the control (A) and the variant (B).
- Choose a two-tailed test unless you fixed a one-sided hypothesis before the experiment started.
- Pick α. 0.05 is the standard (95% confidence).
- Read the verdict, p-value, uplift and confidence interval. Treat a non-significant result as “not enough evidence”, not as proof of no effect.
Statistical Significance Formula
Pooled proportion
z statistic
Confidence interval for the difference
Relative uplift
Statistical Significance Variables Explained
| Symbol | Meaning | Notes |
|---|---|---|
| nA, nB | Visitors in each version | |
| xA, xB | Conversions in each version | |
| p̂A, p̂B | Conversion rates | x ÷ n |
| p̂ | Pooled proportion | (xA + xB) ÷ (nA + nB) |
| z | Test statistic | Difference in rates ÷ standard error |
Statistical Significance Worked Example
Using the example values (press “Load example” above to try them):
- Rates: A = 250/5000 = 0.05; B = 300/5000 = 0.06.
- Pooled rate p̂ = (250 + 300) ÷ (5000 + 5000) = 0.055; standard error = 0.00455961.
- z = (p₂ − p₁) ÷ SE = 0.01 ÷ 0.00455961 = 2.193.
- p-value (two) = 0.028295; less than α = 0.05.
Result: Result is statistically significant = Yes; p-value = 0.028295; z statistic = 2.193.
What “statistically significant” does and does not mean
A significant result at α = 0.05 means that, if there were truly no difference between the versions, a gap this large would occur less than 5% of the time. It does not mean there is a 95% chance that B is better, and it says nothing about whether the difference is large enough to matter commercially.
Look at the confidence interval for the size of the effect. A significant uplift of 0.1 percentage points may not justify a change; a non-significant uplift with a wide interval may simply be an underpowered test.
| Result | Interpretation |
|---|---|
| Significant, large uplift | Strong case for shipping the variant |
| Significant, tiny uplift | Real but may not be worth the cost |
| Not significant, wide interval | Inconclusive; collect more data |
| Not significant, narrow interval around 0 | Good evidence of no meaningful difference |
One-tailed or two-tailed?
A two-tailed test detects a difference in either direction and is the safe default. A one-tailed test has more power for the direction you chose but cannot detect a harmful effect. Use it only when the direction was fixed in advance and an effect in the other direction would lead to the same action as no effect.
How much data do you need?
The required sample size depends on your baseline conversion rate, the smallest uplift you care about, α and the power you want (80% is typical). Smaller effects need far more traffic. Use the sample size calculator before launching an experiment.
Where Statistical Significance Is Used
A/B tests
Checking whether a new page, email or button changed the conversion rate.
Comparing two proportions
Testing any pair of rates, such as response rates in two groups.
Reporting uplift
Giving the relative uplift with a confidence interval, not just a p-value.
Judging power
Seeing whether the test had enough data to detect the effect.
Common Statistical Significance Mistakes to Avoid
- Peeking: stopping the test the first time p dips below 0.05 inflates false positives dramatically. Fix the sample size in advance or use sequential methods.
- Multiple variants or metrics: each extra comparison raises the chance of a false win; adjust α (for example divide by the number of comparisons).
- Unequal traffic split or novelty effects can mislead; run the test across full weekly cycles.
- Too little data: small samples can only detect big effects. Check power and plan sample size first.
- Counting the same user more than once breaks independence.
- Decide the sample size and run time before starting.
- Use α = 0.05 and 80% power as a minimum standard.
- Report the interval for the difference, not only the p-value.
- This test assumes independent visitors and reasonably large counts (at least about 10 conversions and non-conversions per group).
Statistical Significance Terms and Definitions
- Uplift
- Relative improvement of the variant over the control.
- Power
- Probability of detecting a real effect of a given size.
- Type I error
- Declaring a difference when none exists (false positive).
Statistical Significance FAQs
How do I know if my A/B test is statistically significant?
Compare the p-value with your α. If p is below α, usually 0.05, the difference is statistically significant.
What is a good significance level?
0.05 is the convention. Use 0.01 when a false positive is costly and 0.10 for early exploratory tests.
What does 95% statistical confidence mean in A/B testing?
Many tools define it as 1 − p. Strictly, it means the p-value is below 0.05; it is not the probability that the variant is better.
Can I stop my test as soon as it is significant?
Not safely. Repeatedly checking inflates the false-positive rate. Decide the sample size in advance or use a sequential testing method.
What if my result is not significant?
It means the data do not give enough evidence of a difference. Check the confidence interval and power; you may need more data.
What sample size do I need?
It depends on the baseline rate and the smallest uplift worth detecting. Smaller effects need much more data. Use a sample size calculator.
Is this test valid for small numbers of conversions?
The z-test is an approximation that needs roughly 10 or more conversions and non-conversions in each group. For very small counts, use Fisher’s exact test.
Why shouldn't I stop an A/B test as soon as it turns significant?
Checking repeatedly and stopping at the first p < 0.05 inflates the false-positive rate. Decide the sample size in advance, for example with the sample size calculator, and judge the result once that many visitors have been collected.
Is statistical significance the same as a meaningful improvement?
No. With enough traffic a tiny uplift can be significant. Look at the confidence interval and the effect size to judge whether the difference matters for your goals.