Shopify A/B testing: Rollouts, apps and theme split tests

Shopify A/B testing apps and what they cost

Most Shopify testing apps add what Rollouts lacks: revenue reporting, template-level tests and price tests. The prices below are the entry plans shown on each Shopify App Store listing on October 11, 2026. Most scale with your traffic or order volume, so check the tier that matches your store.

Shoplift starts at $99/month for Core (template, theme and URL tests with Bayesian analysis), is priced on site-wide monthly visitors, and puts price testing on Advanced at $399/month. Intelligems starts at $69/month for Smart Content, scales with order volume, and includes price testing in its Unlimited plan at $349/month. Elevate starts at $49/month with template, theme, split URL and visual-editor tests, and adds price tests on Standard at $99/month. ABConvert starts at $99/month with 1,000 test orders included, puts price tests on Growth at $199/month, and states that checkout content, cart and upsell tests need Shopify Plus. Shogun's separate A/B testing app starts at $39/month for 3 active tests, with price testing on Unlimited at $499/month.

If price testing is the goal, one of these apps is the right tool. Shopify's native Experiments can publish a catalog's prices to the treatment group, but Shopify warns that the store can then show different prices than Google and your other sales channels. For tools beyond Shopify, see our A/B testing tools comparison, and for one Shopify-focused alternative, read Otter vs Mida. We make Otter, so check every price here against the vendor's own page.

How Shopify A/B testing works with Otter

Otter has no app in the Shopify App Store. You install it by hand: paste one snippet into layout/theme.liquid right after the opening <head> tag. The snippet loads Otter's script asynchronously and hides the page until test settings arrive, with a 3-second failsafe, so shoppers don't see the original before a variant. The script is about 32 KB gzipped. Because the snippet lives in your theme, Otter runs on every storefront page the theme renders: home, collections, products, search, cart and content pages.

Purchases come from a second piece of code. Otter generates a Custom Pixel that you paste under Settings > Customer events. It listens for Shopify's checkout_completed event and sends a purchase with the order ID, currency and order total. It also sends page views, product views, searches, add-to-cart, checkout started and payment info submitted. Otter joins those events to the visitor's variant through its first-party cookie and ignores duplicate orders by order ID. One limit: Otter only provides the pixel for projects in opt-out consent mode, because Shopify runs custom pixels in a sandbox that can't inherit your theme's consent banner. A store in opt-in mode gets no automatic purchase or revenue tracking from Otter.

With the pixel connected, every variant shows total revenue, revenue per visitor and average order value next to conversion rate, and revenue per visitor gets its own significance test. What Otter can't do matters just as much. It doesn't run on Shopify's checkout pages, so it can observe checkout steps but not change them, and it can't change the price a customer is charged. Starter costs $39/month for one store with unlimited visitors and tests; see Otter pricing for the other plans.

How to split test two Shopify themes

Otter has no Shopify-specific theme test type. A theme split test runs as a split URL test that uses Shopify's theme preview parameter. Set the test URL to a site-wide wildcard such as https://yourstore.com/*, set the variant URL to https://yourstore.com/?preview_theme_id=123 with your unpublished theme's ID, and turn on relay parameters. Otter then keeps each shopper on the page they asked for: a visitor landing on /products/jacket goes to /products/jacket?preview_theme_id=123, not the homepage. Install the Otter snippet in both themes so assignment and conversions carry across.

Two cautions. First, how a real shopper experiences preview_theme_id, including Shopify's preview bar and preview cookies, is Shopify's behavior, not Otter's, so open the variant yourself on desktop and mobile and click through to checkout before launch. Second, a duplicated theme is a snapshot. If someone edits the live theme, changes an app embed or updates theme settings mid-test, the two themes drift apart in ways the test never meant to measure. Freeze theme edits until the test ends.

Shopify's native Experiments now replace a whole theme without a URL parameter, which is the cleaner route if you're on Grow or higher and conversion rate is enough to decide. Otter's approach suits stores that want revenue per visitor and AOV for each theme, more than one challenger, or a Bayesian readout.

What to test first on a Shopify store

Start where buying decisions happen and traffic is high. On product pages, test the order of images, where shipping and returns information sits, and trust badges near the Add to cart button. Baymard Institute lists extra costs (40%) and slow delivery (20%) among the top reasons shoppers abandon carts, so showing delivery times and costs earlier is a reasonable first hypothesis. On collection pages, test product card details and how many products show per row on mobile.

Use revenue per visitor as the primary goal for anything close to the buy button. A change can lift add-to-cart rate while nudging shoppers toward cheaper products, and conversion rate alone hides that. Then check whether your traffic can support the test: a store converting at 2% needs roughly 81,000 visitors per variant to detect a 10% relative lift at 95% significance and 80% power. Run your own numbers in the sample size calculator, and see the ecommerce A/B testing guide for more store test ideas.

Some ideas belong elsewhere. Shipping thresholds, discount mechanics and charged prices live in Shopify settings, not in the page, so test them with Rollouts or a dedicated app. Otter can change how you describe a free-shipping threshold, not the threshold itself.

Shopify A/B testing and SEO

Google's guidance on website testing and search is short. Don't cloak: never show Googlebot different content from what people see. When a test sends people to a different URL, use a temporary 302 redirect rather than a 301; Google says JavaScript redirects are fine too. Put a rel="canonical" tag on alternate URLs pointing to the original, which Google prefers over noindex. End the test once you have an answer and remove the test code.

Most Otter tests on Shopify change the page in place with JavaScript, which Google lists as a valid way to test without new URLs, and small changes such as button text often have little or no effect on rankings. Theme split tests redirect in the browser to the same path with a query parameter added. Otter doesn't assign known bots to a variant, so crawlers get your published theme. One trap applies to native Experiments instead: a catalog experiment can make your store show different prices than Google and other channels, so think about product feeds before testing catalogs.

Rollouts, an app or Otter: which fits your store

Use Shopify Rollouts if you're on Grow or higher, want to test a whole theme, a checkout configuration or a discount, and conversion rate is enough to decide. It costs nothing beyond your plan. Use an app such as Intelligems or Shoplift if you need price, shipping or checkout tests, which Otter can't run. Use Otter if you want revenue per visitor and AOV per variant on theme and content tests, frequentist or Bayesian statistics at the threshold you choose, GA4 events as goals, and a flat price of $39, $79 or $149 a month by number of projects, with unlimited visitors and a 14-day trial that needs no credit card.

Many stores combine them: Rollouts or an app for checkout and pricing, Otter for storefront content and theme tests. If you also run a WooCommerce store, the WordPress A/B testing guide covers Otter's plugin, which tracks WooCommerce revenue without a pixel.