Ecommerce A/B testing: measure revenue per visitor, not just conversions

Native A/B testing on ecommerce platforms

Among Shopify, WooCommerce and BigCommerce, only Shopify documents built-in A/B testing. Its Rollouts feature launched on March 31, 2026, and the Experiment type, on the Grow plan or higher, can test themes, checkout and accounts configurations, catalogs and discounts. Experiments report conversion rate, bounce rate, reached-checkout rate and add-to-cart rate, but not revenue or average order value. Our Shopify A/B testing guide covers Rollouts and the Shopify apps in detail.

Neither WooCommerce nor BigCommerce documents a built-in A/B testing feature, so stores on those platforms use a plugin or a script-based tool. BigCommerce publishes an article on A/B testing as a practice, not a testing feature. Headless and custom stores face the same choice, with the extra option of building assignment into their own code.

Shopify apps such as Intelligems (from $69/month) and Shoplift (from $99/month) go further than Otter in one direction: they can test prices, shipping rates and, on Shopify Plus, checkout content. Those prices are from their Shopify App Store listings on October 11, 2026, and both scale with store size. For a broader comparison that includes Otter, which we make, see our A/B testing tools guide.

How Otter tracks ecommerce revenue

Otter needs to know when an order happens and what it was worth. On Shopify, Otter generates a Custom Pixel that listens for completed checkouts and sends the order ID, currency and total; it's available only for projects in opt-out consent mode. On WooCommerce, Otter's WordPress plugin sends the purchase from the order-received page using WooCommerce's own order data, plus product views, add-to-cart and checkout started. On any other store, including headless storefronts and custom builds, call optimo.revenue() on the confirmation page with the order value, currency and order ID.

Two safeguards keep the numbers clean. Otter ignores a second purchase with the same order ID, so a refreshed confirmation page or a retried request doesn't double-count. Stores that sell in several currencies get revenue converted into one reporting currency using exchange rates refreshed every 24 hours; if a rate or currency is missing, Otter withholds the revenue figure and says why rather than guessing. If you switch it on, Otter also sends exposures, conversions and revenue to GA4 and the GTM dataLayer.

Purchase tracking works with every test type: visual-editor changes, split URL tests between two page designs, targeted tests for a segment such as mobile visitors from paid search, and, on WordPress, whole-theme tests.

What to test first in an online store

Baymard Institute's cart abandonment research, last updated September 22, 2025, puts the average documented abandonment rate at 70.22% across 50 studies. Excluding shoppers who were just browsing, the top reasons were extra costs being too high (40%), slow delivery (20%), not trusting the site with card details (19%), forced account creation (18%) and a checkout that was too long or complicated (17%). Several of those can be addressed before checkout, which makes them good tests for a storefront tool.

Show shipping costs and delivery times on the product page instead of revealing them at checkout. Put returns information and payment trust signals near the buy button. Test product image order, how much of the description shows above the fold on mobile, and whether reviews sit near the price. On collection pages, test the product card: price display, color swatches, review stars and badges. In the cart, test cross-sell placement and free-shipping progress messages. For a structured list, work through the ecommerce CRO checklist, then rank your ideas with the test prioritization calculator.

What Otter can't test in a store

Otter runs in the browser on pages where its snippet loads, which sets clear limits. On Shopify it doesn't run on checkout pages, so it can measure checkout steps but not change them. On any platform it can't change the price a customer is charged. A visual change could alter the price text on a product page, but don't do it: shoppers would see one price on the product page and another at checkout. For price, shipping-rate and checkout tests on Shopify, use a dedicated app or Shopify's own discount and catalog Experiments.

Store-wide settings work the same way. Free-shipping thresholds, discount rules and tax display live in your platform's settings, so Otter can test how you describe them, not the rules themselves. On WooCommerce the checkout is a WordPress page where Otter's snippet loads, so you can test the copy and reassurance around the checkout form, while the order logic stays with WooCommerce.

How much traffic ecommerce A/B testing needs

Orders are rarer than clicks, so store tests need more visitors than most people expect. A store converting at 2% needs roughly 81,000 visitors per variant to detect a 10% relative lift in conversion rate at 95% significance and 80% power. Detecting a 20% lift needs about 21,000. Revenue per visitor varies more from shopper to shopper than a yes-or-no conversion, so RPV tests usually need more traffic again. Check your own numbers with the sample size calculator before launch.

Three habits help. Run tests in whole weeks, because weekday and weekend shoppers behave differently. Don't stop a frequentist test the moment it crosses the line: Otter's estimate tells you how long the planned sample takes, and Bayesian mode suits teams that want to watch results as they come in. For short promotions that end before a classic test could finish, a multi-armed bandit sends 80% of new traffic to the current leader after a 50-visitor warm-up per variant, trading a clean readout for more revenue during the promotion. Otter's Trust Check also flags a sample ratio mismatch, a sign that something in the setup is losing visitors from one variant.

Ecommerce A/B testing and SEO

Product and category pages often earn most of a store's organic traffic, so follow Google's testing guidance. Don't show Googlebot different content from shoppers. When a test redirects to a different URL, use a 302 or a JavaScript redirect, and give the alternate URL a rel="canonical" tag pointing at the original. Finish tests once you have an answer and remove the losing version, because Google may treat an experiment left running for a long time as an attempt to deceive.

Most store tests don't need new URLs at all. Changing a product page in place with JavaScript, as Otter's visual editor does, is a method Google describes as valid, and small layout changes often have little or no effect on rankings. Be careful with anything that changes the prices some visitors see: Shopify warns that a catalog Experiment can make a store show different prices than Google and other channels.