Atchoo vs Shopify Rollouts: Native A/B Testing

Shopify now ships its own A/B testing, so this is the comparison a lot of merchants will reach for first. It’s delivered through a feature called Rollouts, it’s free and built into the admin, and it can run genuine A/B experiments. Atchoo! is a paid third-party app that tests one thing, product page templates, judged on add-to-cart rate. The two overlap less than you’d expect, and for a good number of stores the honest answer is “use both”. Here’s how they actually differ.

TL;DR verdict

Short version: Rollouts is the free, native way to A/B test a whole theme or a checkout/account configuration. Atchoo is the granular way to test one product page template against another and read the result on add-to-cart rate. Different granularity, and genuinely complementary.

What Shopify Rollouts is

Rollouts is Shopify’s native mechanism for shipping changes in a controlled way. You’ll find it in the admin under Markets, and it arrived as part of the Winter ‘26 edition. It comes in two flavours:

• Launch rollouts send a change to all of your traffic at once, with no split. This is a controlled deployment, not an experiment.
• Experiments are the A/B testing type: Shopify serves a treatment against a control, splits traffic between them, and reports which performed better.

When people say “Shopify has native A/B testing now”, the Experiments type is what they mean.

What can an Experiment cover? Two things:

1. Theme changes. You can test an edited version of a theme, or test an entirely different theme, against your current one.
2. Checkout and customer-account configurations. You can test replacement checkout and account configurations against your existing setup.

That’s the scope. It’s theme-level and checkout/account-level testing, run natively, with nothing to install.

Plan and cost

Rollouts itself is available from the Basic plan upward, and it’s genuinely free, it’s included with Shopify, not a paid add-on. The one thing to know: the Experiments type (the A/B testing part) requires the Grow plan or higher. Market-specific targeting needs Advanced or Plus. So a store on Basic can use launch rollouts, but running an A/B experiment means being on Grow or above. Check your plan before you count on it.

One thing Rollouts does not do is price or discount testing, its experiments cover theme and checkout/account configurations, not what you charge. If pricing is your question, neither Rollouts nor Atchoo is the tool; something like Intelligems is the right category, and it’s worth reading our note on price testing risks first.

What Atchoo does

Atchoo! A/B tests product page templates inside your existing theme:

1. You create an alternate product template using Shopify’s native template suffixes (e.g. product.variant-b), no theme duplication, no code.
2. Each visitor is deterministically assigned to a variant via a first-party cookie, so the same visitor always sees the same version. Traffic splits 50/50 by default (configurable).
3. Variant visitors see the alternate template through Shopify’s native template view mechanism.
4. Product views and add-to-cart events are tracked in real time through a Shopify app proxy, first-party, no third-party cookies, with a traffic-source/UTM breakdown.
5. Bayesian analysis calls the winner in plain terms (“there’s a 96% probability variant B is better”) rather than a p-value.

The metric focus is add-to-cart rate, because it’s the closest thing a product page change can directly move, and it accumulates far faster than orders do. Atchoo does not do price testing, checkout testing, sitewide testing, personalization, or heatmaps. It does one job.

The real difference: granularity

The cleanest way to think about these two is the size of the thing you’re testing.

Rollouts works at the theme level. An experiment compares a whole theme (or a whole checkout/account configuration) against another. That’s powerful when your hypothesis is big: “does this redesigned theme beat our current one?” or “does this new checkout configuration convert better?” The unit of the test is large, and Shopify handles the split and the reporting natively.

Atchoo works at the product-template level. A test compares one product page template against another, head-to-head, with add-to-cart rate as the read-out. The unit is small and specific: gallery treatment, section order, trust-signal placement, description layout, CTA position on the product page. You’re isolating a change on the page type where most stores’ highest-value layout hypotheses live.

Neither granularity is “better”. They answer different questions. A whole-theme experiment tells you the redesigned bundle won or lost; it’s less good at telling you which change drove the result. A single-template test tells you exactly which product page treatment moved add-to-cart, but it can’t speak to your header, home page, or checkout.

Analytics and reading results

Rollouts reports through Shopify’s own experiment interface, native to the platform and tied to Shopify’s data.

Atchoo reports on one question in depth: which template gets more visitors to add to cart? Events are tracked first-party in real time, broken down by traffic source and UTM so you can see when a variant wins with paid traffic but not organic, and winners are called with a Bayesian engine aimed at non-statisticians. If the Bayesian approach is new to you, we explain the reasoning in Bayesian vs frequentist A/B testing.

For lower-traffic stores, the metric choice matters a lot: theme and checkout experiments are usually judged on order-level outcomes, and orders are scarce when sessions are low, so those tests take longer to conclude. A product-page test on add-to-cart uses a far more frequent event, which is why it often reaches a decision sooner. More on that in A/B testing low-traffic stores.

Can they work together?

Yes, and for many stores that’s the sensible setup. They test different units through different mechanisms, so there’s a natural division of labour:

• Use Rollouts for the big, native experiments: a theme redesign, a new checkout or account configuration. It’s free and there’s nothing to install.
• Use Atchoo for granular product page work: iterating on layout, imagery, copy, and trust signals, and reading each test on add-to-cart rate.

The one rule is the same as with any pair of testing tools: don’t run a theme-level experiment and a product-template test on the same pages at the same time, or you won’t be able to attribute the result to either change.

Who should pick which

Use Shopify Rollouts if:

• You want free, native A/B testing with nothing to install.
• Your hypothesis is theme-level or checkout/account-level (“does this redesign or this new checkout win?”).
• You’re on the Grow plan or higher (required for the Experiments type).

Use Atchoo if:

• Your testing roadmap is product-page-focused: layout, gallery, copy, CTAs, trust signals.
• You want to isolate a single product template change and read it on add-to-cart rate.
• You want first-party UTM breakdowns and Bayesian winner calls tuned for product page tests.
• You’re on any Shopify plan, including those below Grow.

Frequently asked questions

Does Shopify really have built-in A/B testing now?

Yes. It’s delivered through Rollouts (in the admin under Markets), and the Experiments type runs true A/B tests. It’s free and native, though the Experiments type requires the Grow plan or higher.

Can Rollouts test a single product page layout?

Rollouts experiments are described at the theme level (a whole theme) and the checkout/account-configuration level. Atchoo is built specifically for head-to-head tests of one product page template against another, which is a finer granularity than a whole-theme experiment.

Do I have to choose one or the other?

No. Because they test different units, plenty of stores run theme and checkout experiments in Rollouts and product-page template tests in Atchoo. Just don’t overlap them on the same pages during the same period.

Is Atchoo worth paying for when Rollouts is free?

It depends on what you’re testing. If your work is theme-level or checkout-level, Rollouts covers it for free and Atchoo isn’t the tool. If your work is granular product page testing on add-to-cart rate, that’s the specific job Atchoo is built for, and it’s what Rollouts doesn’t focus on.

The bottom line

Shopify Rollouts is a real, welcome addition: free, native A/B testing for whole themes and for checkout and account configurations, available from the Grow plan up. If those are your questions, use it, there’s no reason to pay for what the platform now gives you. Atchoo sits at a different granularity: single product page templates, tested head-to-head and judged on add-to-cart rate, on any plan. For most stores the two are complementary rather than competing. For the wider field, see our roundup of the best Shopify A/B testing apps and the complete guide to Shopify A/B testing.

If granular product page testing is what you need alongside (or instead of) Rollouts, you can try Atchoo! free for 14 days on the Pro plan: create an alternate template, split your traffic, and watch real add-to-cart data arrive within days. Details on the pricing page.