The hard part of A/B testing isn’t running the test, any decent tool handles that. The hard part is the moment the results come in and a very persuasive voice in your head starts explaining why the numbers mean what you wanted them to mean. Richard Feynman’s line applies perfectly here: the first principle is that you must not fool yourself, and you are the easiest person to fool.

This article is a practical framework for reading test results honestly: what to decide before the test starts, how to handle segments without torturing the data, why winning results are usually smaller than they look, when a rerun is justified, and how to make sure a test teaches you something even when it “loses.”

Step zero: decide everything before you launch

Almost every analysis sin becomes impossible if you write three things down before the test starts. Researchers call this pre-registration, and you don’t need a lab to do it, a note in a spreadsheet is fine:

  1. The hypothesis. Not “let’s try a new layout” but a falsifiable sentence: “Moving the reviews section above the fold will increase add-to-cart rate, because visitors currently bounce before seeing social proof.” A real hypothesis names the change, the expected effect, and the reasoning, so the result can confirm or contradict the reasoning, not just the change.

  2. The primary metric. One of them. For most product page tests that’s add-to-cart rate, it’s the closest metric to the change you made, and it accumulates data fastest. (We’ve written about why add-to-cart rate usually beats conversion rate as a primary metric.) You can track secondary metrics for context, but the win/lose verdict comes from the one you named in advance.

  3. The stopping rule. How long the test runs or how many visitors it needs, decided up front from a sample size and duration estimate, not “until the result looks good.” Peeking at results daily and stopping the moment your variant is ahead is the single most common way merchants manufacture false winners.

If you didn’t do this before your current test: fine, but do the honest version now. Write down what you would have pre-registered, and treat anything outside it as exploratory.

Reading the headline result

When the test hits its planned end, look at the primary metric first and only:

  • A clear result on the primary metric (significant in a frequentist tool, or a high probability-to-beat-control with a meaningful effect in a Bayesian one) is your verdict. Our guide to statistical significance covers what those numbers actually mean, and if you’re curious why tools like Atchoo! report “probability B beats A” instead of p-values, see Bayesian vs frequentist testing.
  • An inconclusive result is a result. “We couldn’t detect a difference at this sample size” is real information, it usually means the change didn’t matter much either way, and you can move on to a bolder idea.
  • Resist the upgrade urge. If add-to-cart was flat but revenue-per-visitor looks up 8%, do not quietly promote revenue to primary metric. Downstream metrics are noisier, and if you scan enough of them, one will always look good by chance. That’s not analysis; that’s shopping for a verdict.

Segments: hypothesis generators, not verdicts

Every dashboard tempts you to slice results by device, traffic source, and new versus returning visitors. Slicing is genuinely useful, and genuinely dangerous, so hold two rules:

Rule 1: segment findings are questions, not answers. Each segment has a fraction of your total sample, so segment-level numbers are far noisier than the headline. And every extra slice you examine is another lottery ticket for a fluke. Check enough segments and something will show a dramatic swing by pure chance, this is the multiple comparisons problem, and it doesn’t care how plausible the story you attach to it sounds.

Rule 2: the overall result is the verdict; segments explain and inspire. The right use of segments looks like this:

  • Device: if the variant wins on desktop but loses on mobile, don’t ship “desktop-only” off one test. Instead ask why, did the new layout push the buy button below the fold on small screens? That’s a QA insight and a new hypothesis: fix the mobile layout, rerun. (Given most Shopify traffic is mobile, the device segment is usually the one most worth understanding.)
  • Traffic source: a variant that wins with paid social traffic but not organic might be resonating with cold visitors specifically. Interesting! Now design a test that targets that idea deliberately.
  • New vs returning: returning visitors have seen your old page and may react to change itself; new visitors are the cleaner read on the design. A big gap here is a reason to let the test run longer or to weight your interpretation, not to declare two different winners.

A segment finding graduates to “true” only when you test it as its own pre-registered hypothesis and it wins on its own.

The winner’s curse: your lift is smaller than it looks

Here’s the uncomfortable statistical fact almost nobody prices in: the measured lift of a winning test is, on average, an overestimate of the true lift.

Why? Your measured result is true effect plus random noise. Tests cross the “winner” threshold more easily when the noise happened to break in their favour, so the set of tests you declare winners is enriched with lucky overestimates. This is a cousin of the winner’s curse from auction theory, driven by plain regression to the mean: extreme measurements tend to be followed by less extreme ones.

Practical consequences:

  • Discount the headline number. If the test says +18% add-to-cart, ship it, but plan around something more modest. The smaller your sample and the more marginal the significance, the bigger the haircut.
  • Don’t stack lifts. Five winners at +10% each do not make your store 61% better. Some of each lift was noise, and effects overlap. If your analytics don’t show the compounded miracle, the tests weren’t “wrong”, the arithmetic was.
  • Expect post-launch reality checks to come in low. A winner that shows a smaller effect after full rollout is the norm, not a scandal.

When to rerun a test

Rerunning is legitimate in some cases and result-shopping in others. Fair reasons to rerun:

  • The test was contaminated. A site redesign mid-test, a broken variant you discovered late, a flash sale that flooded one week with unrepresentative traffic. Fix the issue, rerun clean.
  • The sample was far short of plan because traffic underdelivered, an inconclusive result from an underpowered test says little. Rerun with a longer window, or accept that low-traffic stores need bolder tests.
  • Seasonality mismatch. A test run entirely inside a discount event may not generalise. Validate the winner in a normal period before treating it as permanent truth.
  • A surprising result you’re about to act on expensively. Replication before a big rollout is good practice, not cowardice.

The illegitimate reason: “it lost and I really liked the variant.” Rerunning until you get the answer you wanted is just slow-motion p-hacking. If you rerun, pre-register again, and let the replication be the verdict.

Document the learning, not just the outcome

A test’s outcome is ship/don’t-ship. A test’s learning is what it told you about your customers, and that’s the compounding asset. For every test, record in a simple log:

  • The hypothesis and reasoning (copy it from your pre-registration)
  • Dates, sample size, and any anomalies during the run
  • Primary metric result, with the actual numbers
  • Verdict: shipped / not shipped / rerun planned
  • What we now believe about our customers, one or two sentences. “Social proof placement matters more on mobile than desktop.” “Our visitors don’t read below the fold.”
  • Follow-up hypotheses the test generated (often from segments, see above)

This log is what stops your team retesting the same idea in 18 months, and it turns losing tests into paid-for knowledge. A “loser” that kills a widely held internal belief is often worth more than a 5% winner.

Atchoo’s results screen gives you the Bayesian verdict and the device and traffic-source breakdowns; the honest interpretation, primary metric first, segments as questions, lifts discounted, is on you, and now you have the framework.

If you’re earlier in the journey, the complete guide to Shopify A/B testing covers the full process from hypothesis to rollout.

Frequently asked questions

My variant is winning after four days. Can I stop the test?

Almost certainly not. Early leads flip constantly, small samples are dominated by noise, and days of the week have different traffic quality. Stopping the moment you’re ahead is the most reliable way to ship a false winner. Honour the stopping rule you set before launch, and run whole weeks.

The overall result was flat but one segment shows a big win. Should I ship for that segment?

Not off this test alone. With multiple segments examined, a standout segment is expected by chance. Treat it as a hypothesis: design a follow-up test aimed at that segment’s insight, pre-register it, and let the replication decide.

How much should I discount a winning result?

There’s no universal formula, but the principle is: the smaller the sample and the more borderline the result, the bigger the haircut. A comfortable winner on a large sample might be close to its measured lift; a just-barely winner on a small sample could easily be half of it or less. Plan conservatively and check post-launch metrics.

What should I do with an inconclusive test?

Log it, learn from it, and move to a bolder hypothesis. Inconclusive usually means the change was too small for your traffic to detect, which itself tells you that element probably isn’t where your big wins live.