Your test dashboard says variant B is up 14% after five days, and every instinct says “ship it”. Statistical significance exists to answer one awkward question before you do: could a gap that size have appeared by pure luck? Very often the honest answer is yes, and understanding why is the most useful piece of statistics a merchant will ever learn.
What statistical significance actually is
An A/B test splits your visitors into two groups at random. Even if both groups see the identical page, their conversion rates will not match. One group happens to catch a few more ready-to-buy visitors, the other catches a few more browsers, and the numbers drift apart for no reason at all.
Statistical significance is a ruler for that randomness. It asks: is the difference I’m seeing bigger than the differences chance alone routinely produces? If yes, you have evidence of a real effect. If no, you have noise, however exciting the percentage looks.
That’s the whole idea. Everything else, p-values, confidence levels, one-tailed tests, is machinery for making that judgement precisely instead of by gut feel.
The coin-flip version
Flip a fair coin 10 times and you’ll get 7 or more heads about 17% of the time. Nobody would conclude the coin is rigged from that, 10 flips is simply too few to tell a biased coin from a fair one.
Now flip it 1,000 times and get 700 heads. That’s effectively impossible for a fair coin, so you’d rightly conclude something real is going on.
Your A/B test is the same experiment wearing a different outfit. “Variant B converted 7 visitors out of 100 while A converted 5” is the 10-flip situation: suggestive, proof of nothing. Significance testing tells you which situation you’re actually in.
P-values in plain English
The p-value answers this question:
If there were truly no difference between A and B, how often would random chance alone produce a gap at least this big?
A p-value of 0.30 means: even if your new page were exactly as good as the old one, you’d see a gap like this about 30% of the time anyway. Weak evidence. A p-value of 0.01 means chance would produce this gap only 1 time in 100, much harder to dismiss.
Two things a p-value is not:
- It is not the probability that B is better than A. A p-value of 0.04 does not mean “96% chance B wins”. It’s a statement about how surprising your data would be in a world where nothing changed. (If you want a direct “probability B beats A” number, that’s the Bayesian approach, a genuinely different calculation.)
- It is not a measure of how big the effect is. With enough traffic, a trivial 0.5% lift can be highly significant. Significance says “probably real”, not “worth caring about”.
What “95% confidence” actually means
The convention in most testing tools is to call a result significant when the p-value drops below 0.05, expressed as “95% confidence” or a “5% significance level”.
Here’s what that buys you: if you only ever ship winners at 95% confidence, then in tests where there’s genuinely no difference, you’ll wrongly declare a winner about 1 time in 20.
That’s it. It is not:
- a 95% probability that B beats A,
- a guarantee the lift is real,
- a promise the measured lift (say, +14%) is the lift you’ll keep. Results that just scrape over the significance line tend to overstate the true effect, because tests that got lucky are more likely to cross the line early.
Why 95% and not something else? It’s a trade-off, not a law of nature. Demand 99% and you’ll need far more traffic; accept 90% and you’ll ship more false winners. For most stores, 95% is a sensible default; 90% can be a reasonable, eyes-open choice for low-traffic stores willing to accept more risk.
Why “it looks like it’s winning” is not evidence
Here’s a concrete example. Suppose your control converts visitors to add-to-cart at 8.0% and your variant is showing 9.2%, a 15% relative lift.
- With 1,000 visitors per variant (80 vs 92 add-to-carts), the p-value works out to roughly 0.34. A gap this size appears by chance about a third of the time when nothing has changed. This result means almost nothing.
- With 10,000 visitors per variant (800 vs 920 add-to-carts), the same rates give a p-value around 0.003. Now chance is an implausible explanation.
Identical percentages, opposite conclusions. The lift you see on a dashboard carries no information about whether it’s real, only the combination of lift and sample size does. This is why eyeballing a bar chart, or trusting a three-day-old test because “the trend is clear”, routinely leads merchants to ship changes that do nothing (or quietly hurt).
You can check any result yourself in about ten seconds with our free A/B test significance calculator, paste in visitors and conversions for each variant and it does the maths.
The peeking problem
This one catches almost everybody.
Suppose you check your test every morning and plan to stop the moment it shows 95% significance. Sounds diligent. It’s actually a statistical trap: that 5% false-positive guarantee only holds if you check once, at a sample size you fixed in advance.
Why? Random noise wobbles. Over a few weeks of daily peeks, a test where nothing is happening will at some point wobble across the significance line just by chance, and if you’re poised to stop the instant it does, you’ll catch the wobble and call it a winner. Check often enough and your real false-positive rate can climb from 5% to 25% or beyond. Evan Miller’s classic essay How Not to Run an A/B Test walks through the simulations.
The fix is boring but effective:
- Decide your sample size before you start.
- Look at the dashboard as often as you like, but only to check the test is running properly, not to decide whether to stop.
- Make the ship/kill decision once, when the planned sample is in.
(Bayesian methods, which report a running “probability B beats A”, degrade more gracefully under peeking, one reason Atchoo! uses a Bayesian readout for its product-page template tests, but no method makes stopping at the first flattering number completely free.)
One-tailed vs two-tailed, briefly
A two-tailed test asks “is B different from A?”, it can detect a win or a loss. A one-tailed test asks only “is B better?” and reaches significance sooner, because it spends its entire error budget in one direction.
That sounds appealing until a variant underperforms and your one-tailed test is structurally blind to it. For merchants, the losing case is exactly the one you most need to catch, a worse product page costs real money every day it runs. Use two-tailed as your default; treat one-tailed results from any tool as slightly flattering. When a calculator or app doesn’t say which it uses, assume the more optimistic option and add a pinch of scepticism.
What to do instead of guessing
A significance habit that works in practice:
- Before the test: pick the metric (for product pages, add-to-cart rate is usually the right one), estimate the sample you need, and commit to a duration in full weeks.
- During: let it run. Resist the trend line.
- After: run the numbers through a significance calculator or use a tool that reports probability-of-winning directly. Then decide.
If you’d rather not do any of this by hand: Atchoo runs A/B tests on Shopify product page templates and reports results as a plain-English Bayesian probability that the variant beats the control, no p-value interpretation required. There’s a 14-day free trial, and the complete guide to Shopify A/B testing covers the whole workflow from hypothesis to rollout.
Frequently asked questions
Is 90% confidence good enough?
Sometimes. At 90% you’ll ship a false winner about twice as often as at 95%, in exchange for needing meaningfully less traffic. For a low-risk change on a low-traffic store, that can be a rational trade. For anything expensive to reverse, stick with 95%.
My test hit 95% significance on day 3. Can I stop?
Almost certainly not, this is the peeking problem in action. Early significance on a small sample is exactly what random wobble looks like. Run to your planned sample size and at least one full week (ideally two), so weekday and weekend shoppers are both represented.
Can a result be statistically significant but not worth shipping?
Yes. Significance means “probably not chance”, not “big”. A high-traffic store can get a significant +0.3% lift that’s within measurement error of your costs to maintain the change. Judge results on effect size and significance.
What if my test never reaches significance?
That’s a legitimate outcome: the change probably doesn’t matter much either way, at least at a size you can detect. Ship whichever version you prefer for other reasons, log the learning, and test something bolder next, bigger swings need far less traffic to detect.