Every A/B testing tool sits in one of two statistical camps, and the camp it picks changes what the numbers on your dashboard actually mean. One reports p-values and “significance”; the other reports “probability that B beats A”. Both are legitimate. They are not interchangeable, and knowing the difference stops you misreading your own results.
The two schools in one paragraph each
Frequentist statistics is the classical approach behind the p-value. It treats the true conversion rates as fixed but unknown, and asks: if there were really no difference between A and B, how often would chance alone produce data like mine? You set a threshold in advance (usually 95% confidence), collect a pre-planned sample, and get a binary verdict: significant or not.
Bayesian statistics treats the conversion rates themselves as uncertain quantities and updates a probability distribution over them as data arrives. Start with a prior (an assumption about what’s plausible before the test), feed in observed visitors and conversions, and out comes a direct statement: given the data so far, there’s an 88% probability that B’s true rate is higher than A’s.
Same data, two different questions. Frequentist: “how surprising is my data if nothing changed?” Bayesian: “given my data, what’s probably true?"
"Probability B beats A” vs a p-value
This is where merchants get burned, because the two numbers look similar and mean very different things.
A p-value of 0.05 does not mean there’s a 95% chance your variant is better. It means: in a hypothetical world where the variants are identical, data this extreme would show up 5% of the time. It’s a statement about the data, assuming no effect, read our plain-English guide to statistical significance for the full unpacking.
A Bayesian “92% probability B beats A” means what it appears to mean: given everything observed, and the model’s assumptions, there’s a 92% chance the variant’s true add-to-cart rate is higher than the control’s. It answers the merchant’s actual question directly.
Bayesian analysis also gives you a second, underrated number: expected loss (sometimes “risk”), if you ship B and it turns out to be the worse variant, how much conversion rate do you stand to give up, on average? A variant can be 90% likely to win while carrying almost zero downside if wrong, which is often all the justification a busy merchant needs.
The same test, read both ways
Numbers make the contrast concrete. Suppose a product page test has collected 4,000 visitors per variant. The control produced 320 add-to-carts (8.0%); the variant produced 356 (8.9%), an 11% relative lift.
The frequentist readout: a two-proportion test gives a p-value of roughly 0.15. Verdict: not significant. Officially, you’ve learned nothing you’re allowed to act on, even though the variant has led for weeks.
The Bayesian readout on the same data: the probability that the variant’s true rate beats the control’s works out to roughly 92%, and the expected loss if you ship it and you’re wrong is a small fraction of a percentage point of add-to-cart rate.
Neither readout is lying. The frequentist one is correctly refusing to certify the result at the 95% standard; the Bayesian one is correctly describing where the evidence points and what the downside looks like. But notice which one a merchant can actually use: “92% likely better, tiny downside if not” supports a real decision, keep running if the change was costly, ship if it’s easily reversed, where “not significant” reads as a dead end.
Why Bayesian readouts are easier to act on
The numbers say what they look like they say. Nobody misreads “there’s an 88% chance the new template is better, and if it isn’t, you’re probably losing under 0.1% add-to-cart rate”. P-values, by contrast, are so routinely misinterpreted that even researchers who use them daily get the definition wrong.
Every decision maps to a threshold you choose. “Ship when probability-to-win exceeds 95%” or “ship when expected loss drops below a level I don’t care about” are business rules a non-statistician can set and defend.
No cliff edge. A frequentist test at p = 0.06 is officially “not significant”, one at p = 0.04 is “significant”, an absurdly sharp line for numbers that are nearly identical. Bayesian results degrade smoothly: 85% probability-to-win is genuinely useful information, not a failure.
Peeking hurts less. The frequentist 5%-false-positive guarantee assumes you check once, at a pre-planned sample size; checking daily and stopping on green quietly wrecks it. Bayesian probabilities remain interpretable whenever you look. That’s less fragile, not bulletproof, stopping the instant the number flatters you still biases you toward lucky noise under any framework. Whatever your dashboard says, plan a sample size and duration before you start.
The honest case for frequentist testing
It would be dishonest to present this as a knockout, so here’s the other side.
- Guaranteed error rates. Run frequentist tests by the book and your false-positive rate is capped at exactly the level you chose. Bayesian methods make no such promise in those terms.
- Priors are a choice. Bayesian results depend partly on the prior. Sensible tools use weak, neutral priors that the data quickly overwhelms, and with reasonable traffic the influence is negligible, but it’s an assumption, and you should know it’s there.
- A shared standard. “p < 0.05, two-tailed, pre-registered sample” is a lingua franca. If you need to defend a result to a sceptical finance team or run a large experimentation programme with strict false-discovery control, the frequentist toolkit is built for exactly that.
- They agree in the end. Given enough data, both approaches converge on the same answer. The disagreements live at the margins, small samples, early stopping, borderline results, which, inconveniently, is where most real merchant tests live.
Side by side
| Frequentist | Bayesian | |
|---|---|---|
| Core output | p-value / significant-or-not | Probability B beats A, expected loss |
| Answers | ”How surprising is this data if nothing changed?" | "Given this data, what’s probably true?” |
| Reading mid-test | Invalidates guarantees (peeking) | Valid anytime, though early stopping still risks noise |
| Borderline results | Hard cliff at the threshold | Graded evidence |
| Assumptions | Fixed sample, pre-set threshold | Choice of prior |
| Best suited to | Large programmes, formal error control | Practical decisions, smaller samples, non-statisticians |
Where each is appropriate
Frequentist fits when you run many tests with dedicated analysts, need auditable error-rate guarantees, or operate in a context (pharma, academia, big-company experimentation platforms) where p < 0.05 is the required currency.
Bayesian fits when the people reading results are merchants rather than statisticians, traffic is moderate so you want graded evidence instead of a pass/fail cliff, and the goal is a good shipping decision rather than a publishable claim. That describes almost every Shopify store.
This is why Atchoo! reports Bayesian results for its product page template tests: a probability that the variant beats the control on add-to-cart rate, in plain English, rather than a p-value the reader has to translate. The full workflow, from choosing what to test through to rolling out a winner, is in our complete guide to Shopify A/B testing.
Neither school fixes bad experiments
Worth saying plainly: the statistical framework is the last step in the chain, and the least common source of failure. A test with a mushy hypothesis, mixed-up tracking, three days of runtime or too little traffic for the effect it’s chasing will produce garbage under either school, Bayesian garbage just comes with a friendlier label. Get the experiment design right first; then the choice of statistics determines how honestly you can read the result, not whether there’s a result worth reading.
Frequently asked questions
Which one is more accurate?
Neither, in the sense merchants usually mean. With identical, well-collected data both frameworks point to the same practical conclusion in the vast majority of cases. They differ in what question they answer, how they behave mid-test and how easily their outputs are misread, not in some underlying accuracy.
What probability-to-win should I require before shipping?
A common convention is 95%, mirroring the frequentist standard. For cheap-to-reverse changes, some merchants act at 90% and accept the extra risk knowingly. Pair the probability with expected loss: high probability and trivial downside is the comfortable zone.
Does Bayesian testing need less traffic?
Not fundamentally, the information in the data is the same. Bayesian analysis can support a decision sooner because “88% probability, negligible expected loss” is actionable where “not yet significant” isn’t, but detecting small effects reliably needs the same order of traffic either way. Run the numbers with our sample size calculator.
Do I need to set a prior in Atchoo?
No. Atchoo uses neutral priors under the hood, and once a test has accumulated a normal amount of traffic the data dominates the result. You read the probability; the machinery stays out of sight.