How to A/B Test on Shopify With Rollouts: A Native Testing Guide

Learn how to run A/B tests on Shopify with native Rollouts: scheduling, what to test, how long to run a test and how to read the results.

Liquid Gold 2 min read

Shopify’s Rollouts feature brings scheduling and A/B testing into the admin, so you can test changes without a third-party tool. Shopify introduced Rollouts in its Winter ’26 Edition in December 2025 for scheduling and testing storefront changes, and the Spring ’26 Edition in June 2026 extended it so you can publish a new theme, checkout configuration or customer accounts setup at a scheduled time or as an A/B test. Here is how to use it well.

What Rollouts does

  • Scheduling. Set a change to go live at a chosen time, which is useful for launches and sales.
  • A/B testing. Show different versions to different visitors and compare results.

Availability can vary by plan and store, so check your admin for what you can test.

What to test first

Choose changes with a clear hypothesis and meaningful traffic:

  1. Product page layout, such as image gallery position
  2. The headline and offer on the homepage hero
  3. Add-to-cart button copy or placement
  4. Shipping threshold messaging, such as “Free shipping over $75”
  5. Checkout settings that Rollouts supports

How to set up a good test

  1. Write a hypothesis. “Moving reviews above the fold will increase add-to-cart rate.”
  2. Choose one main metric. Conversion rate, revenue per visitor or add-to-cart rate.
  3. Change one thing. If you change five things, you cannot tell which worked.
  4. Check traffic. Low-traffic stores may need longer tests or bigger changes.
  5. Set a duration. Run for at least one to two full weeks to cover weekday and weekend patterns.
  6. Avoid special periods. Black Friday, product launches and big campaigns distort behaviour.

How to read results

  • Wait until the test has enough data before deciding.
  • Compare revenue per visitor, not only conversion rate. A change can lift conversion but reduce order value.
  • Segment by device. A layout that works on desktop may fail on mobile.
  • Be wary of tiny differences. A 1% change on little traffic is probably noise.

Lower the risk of big changes

For bigger changes, such as a new theme, run an A/B test first rather than switching everyone at once. Watch for errors, speed changes and conversion drops before you publish to all visitors.

Pair it with simulation

Shopify’s SimGym tool simulates shopper behaviour to predict outcomes before you launch. It can help shortlist ideas, but real visitor tests are the final proof.

Common mistakes

  • Ending tests early when a variant looks ahead
  • Testing without a hypothesis
  • Running overlapping tests that interfere with each other
  • Forgetting to document what you learned

FAQ

Do I still need a third-party testing tool? Not necessarily. Native Rollouts can cover many tests. Specialist tools may offer more advanced targeting.

How much traffic do I need? It depends on the size of the effect you expect. Smaller effects need more traffic.

Related reading: What Is Agentic Commerce? How Shopify Stores Sell Inside ChatGPT, Gemini and Copilot, Shopify Product Recommendations vs Upsell Apps: Do You Still Need an App? and Shopify Scripts Are Gone.

Want testing handled?

Liquid Gold’s store reviews identify what to test first. Request a store review →

Keep reading

Related articles