# Ab Testing with Google Analytics _2026-11-11_ You've got a variant showing a clear lift in GA4, your boss wants it live by Friday, and nobody has checked whether those extra purchases exist in Shopify or WooCommerce. That's the normal starting point for **A/B testing with Google Analytics** in the UK, and it's where good tests go wrong. GA4 is a measurement layer, not your till. Consent choices, duplicate purchase events and a denominator built on sessions can all make a loser look like a winner. I'd treat every reported win as a candidate, then force it to survive three checks you control before launch: the right revenue metric, enough traffic to detect a real effect, and a reconciliation against actual orders. ## Choosing a Metric That Reflects Revenue A **statistically significant conversion lift** doesn't prove you've made more money. GA4's session key event rate counts sessions containing a purchase, so a variant can add cheap orders while discounts, cancellations and refunds erase the gain. At Great Britain's conversion rate of **roughly 1.94% in Q4 2024, down from about 2.18% in Q4 2023**, according to [PPC Geeks' UK ecommerce funnel analysis](https://ppcgeeks.co.uk/ppc/conversion-funnel-analysis/), a handful of low value orders can easily outweigh the headline rate. Say a free shipping banner lifts orders by **6%** in a hypothetical test but cuts **average order value** by **9%**. Multiply those together and **revenue per visitor falls by about 3.5%**. The banner has a higher conversion rate and should still lose. ![A digital illustration showing a Google Analytics dashboard with conflicting data on sales volume and total revenue.](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/5bc5b027-4fa7-48af-8882-1e021f7db99f/ab-testing-with-google-analytics-data-analysis.jpg) ### Fix the hierarchy before launch Write the hierarchy down and don't swap it after data arrives. I would make **net revenue per eligible visitor** the decision metric because it captures order count and basket value together, then use guardrails to stop a test that wins on revenue by breaking something else. **Metric roles in a UK ecommerce A/B test** | Metric | Role | Why it matters | When to read it | |---|---|---|---| | Revenue per eligible visitor | Primary decision metric | Captures order count and basket value together | At the end of the test | | Purchases per session (GA4 key event) | Session-based check | Matches the GA4 denominator | Alongside revenue, throughout | | Average order value | Guardrail | Shows whether gains come from smaller baskets | Daily during the test | | Refund and cancellation rate | Guardrail | Catches orders that never become revenue | After the refund window closes | | Add-to-cart and checkout start | Secondary diagnostic | Explains where the change acts | Never as the sole basis for a winner | | Checkout errors and page speed | Guardrail | Detects a variant that breaks payment | Daily | Add-to-cart and checkout starts tell you where a change acts. They don't tell you whether to ship it. ## Sample Size and Test Length Before You Launch A UK store on a few thousand sessions a week can look decisive after one noisy week and still be nowhere near a real revenue change. The sample size calculation is what stops that. Use your own recent site data as the baseline conversion rate. A published ecommerce benchmark is the wrong input: it is not your traffic, your consent rate, or your purchase event. ### Four inputs you set in advance Set these before anyone builds a variant. - **Baseline conversion:** Take the primary metric from your own recent GA4 data, not a benchmark. If consent drops a large share of sessions, or a purchase event fires twice, that baseline is already distorted and the sample will be too small. - **Smallest effect worth detecting:** Pick the relative lift that would pay for the build and the upkeep. - **Power:** Use **80%** as the working standard. - **Alpha:** Use a **two-sided 5%** level, so the test can detect harm as well as gain. Run those inputs through [Otter's sample size calculator](https://www.otterab.com/free-tools/sample-size-calculator) before you build anything. Otter recommends that check before any launch, so you know whether the site has enough traffic. Finding the shortfall in week three wastes the variant and the visitors already assigned. **Approximate sessions needed per variant** | Baseline conversion | Smallest effect to detect | Approx. sessions per variant | |---|---|---| | 2% | 10% relative uplift | 80,600 | | 2% | 20% relative uplift | 21,100 | | 2% | 30% relative uplift | 9,800 | | 3% | 10% relative uplift | 53,000 | Those figures are typical outputs for an 80% power, two-sided 5% test on a conversion rate. A revenue-per-visitor goal needs its own baseline, because order value moves more than a yes-or-no purchase, and duplicate purchase events inflate both the rate and the revenue. I would not treat a GA4 lift as ready to ship until the same period's order records move the same way. ### Baseline first, then full weeks Record the same KPI for the **four weeks before** the change, then compare it with an equivalent period afterwards. Run for at least **one complete week**, ideally two, so weekday and weekend behaviour both appear in each arm. Fix the end date before launch. Before you read a result, confirm both arms reached the planned sample in your GA4 comparison. A shortfall is a reason to wait, not a reason to lower the bar. If the traffic maths says six weeks and the business wants an answer in six days, change the test to something with a larger expected effect, or don't test it. ## How to Set Up an A/B Test with GA4, Step by Step GA4 records outcomes but it doesn't split traffic. Your testing tool assigns visitors and changes the page, while GA4 receives the exposure and purchase events you send it. Get that division clear and the setup becomes straightforward. The flow below shows where assignment ends and GA4 measurement begins. ![A four step infographic illustrating the process of conducting A/B testing using Google Analytics 4 tracking.](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/dd08fa55-138e-4370-9286-f5b6043086e1/ab-testing-with-google-analytics-testing-process.jpg) Before step one, decide how GA4 will know which arm a session belongs to. Send the **variant name as an event parameter on every event** and register it as a **custom dimension**, then build the Exploration that compares the arms. Build that comparison before launch, not after, because custom dimensions don't backfill history. Our [GA4 integration documentation](https://www.otterab.com/docs/building-tests/ga4-integration) covers the connection side. 1. **Define one primary outcome.** Pick completed purchases per session or revenue per visitor before you build anything. Two primary outcomes means you'll choose whichever looks better later, which isn't a test. 2. **Confirm the key event.** Find the GA4 event representing that outcome and check it fires **once per order**, with transaction ID, value and currency attached. The common slip is a purchase event that refires when a customer reloads the confirmation page. 3. **Build one change.** Control stays untouched and the variant changes a single element, such as the headline or the delivery message. Change three things and you'll know something worked without knowing what. 4. **Make assignment persistent.** A returning visitor must stay in the same arm, with a random **50/50 split** for new visitors. If someone sees control on Monday and the variant on Tuesday, their purchase belongs to neither arm cleanly. 5. **Send identical events from both arms.** Same event names, same parameters, same currency. Renaming an event in one arm, even by accident in a tag, manufactures a false winner out of missing data. 6. **Validate with real test orders.** Place an order through each arm and watch pageview, exposure and purchase arrive in **DebugView** and Realtime. Skipping this step is how teams lose every purchase from one arm and only notice at the readout. 7. **Fix the sample, start and end dates.** Write them into the test plan before going live. A test without a pre-set stopping point will be stopped at the most flattering moment. ### What good validation catches Check the exposure event fires on assignment, not on page load completion, so slow pages don't undercount one arm. Confirm internal traffic is filtered, preview modes don't pollute the split, and the variant parameter survives the journey from landing page to checkout. On Shopify and other hosted checkouts, that last hop is where parameters get dropped, so follow your own test order all the way to the thank you page and confirm the arm is still attached. Ten minutes here saves a fortnight of arguing about numbers nobody trusts. > If you can't see your own test order in DebugView with the right variant attached, you haven't got a test yet. You've got a hope. ## Reading Lift, Significance and Confidence Intervals **Lift** is the relative difference between arms: (variant rate minus control rate) divided by control rate. It's a percentage of a percentage, which is why small bases produce impressive sounding numbers. Always translate it back into orders and pounds before anyone gets excited. This chart sets out how lift, the p-value and the interval fit together. ### A worked example Say a UK homewares retailer runs a hypothetical test. Control converts **1,840 of 87,600 sessions (2.10%)** and the variant converts **2,023 of 87,600 (2.31%)**, a relative lift of about **10%**. The **95% confidence interval runs from about +3% to +17%**, so the result clears the decision rule, but the true gain could sit anywhere in that range. Budget on the cautious end, not the headline. My decision rule needs both parts: a **p-value below 0.05** and a **confidence interval that excludes zero**. One without the other isn't a positive result. Then report practical significance alongside it, meaning the absolute incremental orders and revenue per month at current traffic. A 10% lift worth a few hundred pounds a month may not justify maintaining a complex variant. The short video below walks through reading these figures on a live result. ### Three rules that protect the result - **No peeking for significance.** Checking daily and stopping at the first significant reading inflates false positives. Read the result at the planned sample. - **No metric switching.** If revenue per visitor was primary, a late surge in add-to-cart doesn't rescue the test. - **Frozen exclusions.** Write the rule for internal traffic, duplicate exposures, payment failures and anomalous bulk orders before launch, then apply it identically to both arms. > **Practical rule:** Report the interval and the monthly cash value in the same sentence as the lift. It stops a wide, weak result being sold internally as a breakthrough. ## Consent, Missing Events and Other Failure Points A UK shopper lands on the variant, the banner has shifted, and they reject analytics. That session never reaches the report. The remaining consented users often convert at a different rate, so the variant can show an apparent lift that is really **differential measurement**. Consent is the most damaging UK failure point because it is invisible in a standard conversion total. Analytics cookies are optional. If your variant changes the page around the banner and more people reject, that arm loses measured sessions and purchases. Google notes Consent Mode has region specific behaviour for the UK and that denied consent users may be measured differently from consented users, as set out in Google's Consent Mode guidance. Report **consent rate per variant**, plus modelled versus observed conversions and exposure counts. Check balance by consent status, region, browser and traffic source. If consent rates diverge, compare consented users only and say so plainly in the readout. The [privacy and consent documentation](https://www.otterab.com/docs/getting-started/privacy-and-consent) explains how to keep that measurement design written down. ### Sample ratio and technical faults Check the **sample ratio** daily. A planned 50/50 split that drifts by several points is an assignment bug, bot traffic, caching or a consent interaction until you prove otherwise. That drift undermines the assumptions behind your significance test. Pause and fix it. Do not analyse around it. Duplicate purchase events and a gap between GA4 revenue and store orders threaten the result in the same way. A reload that fires the purchase event twice inflates GA4 above the ledger. An event on the button, rather than on confirmed payment, records a conversion with no order. I hold any reported winner until those figures are checked against order records. **Symptoms, likely causes and fixes** | Symptom | Likely cause | Fix | |---|---|---| | Arms split more than a few points away from 50/50 | Assignment bug, bot traffic or caching | Pause, check assignment logs, exclude bots and rerun | | Consent rate differs between arms | Banner change alters acceptance | Report consent rate per variant and compare consented users | | GA4 purchases exceed store orders | Duplicate events on confirmation reload | Deduplicate using the transaction ID | | Conversions recorded with no matching order | Event fires on click, not on payment | Move the trigger to the confirmed order | | Conversions up, revenue flat | Smaller baskets or heavier discounts | Make revenue per visitor the primary metric | | One-day spike in one arm | Internal traffic or a bulk order | Apply the frozen exclusion rule | Server side collection can improve resilience when browsers and ad blockers reduce visibility, but it cannot recreate an action that was never captured. Treat GA4 as a diagnostic layer and keep the store ledger as the record of money. That ordering matters most exactly when the two disagree. ## Check the Winner Against Your Order Records Before Rollout A significant result earns a verification pass, not a deployment. Run these five steps in order, using the same dates for every pull. 1. **Pull the store ledger.** Export orders for both arms from Shopify, WooCommerce or your order system for the exact test window. 2. **Compare counts and revenue.** Set GA4 purchases and revenue per arm against those orders. Investigate any gap above a few per cent before you interpret lift, starting with duplicate transaction IDs and missing currency values. 3. **Recheck consent and sample ratio.** Confirm neither arm lost measurement at a higher rate and the split stayed close to plan. 4. **Re-read the hierarchy.** The primary metric, average order value, refund rate and checkout errors must still point the same way. One guardrail moving hard against you is enough to hold. 5. **Roll out and compare.** Ship to everyone, then compare the following four weeks against your pre-test baseline using the same KPI definition. ![A hand-drawn illustration comparing purchase data between Control and Variant groups with GA4 metrics shown.](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/b9956171-5e7e-428c-9732-f51d3fa7bc87/ab-testing-with-google-analytics-data-comparison.jpg) This is the one place a tool saves time. We connect to a GA4 property with one click via OAuth, any existing GA4 event can be used as a test goal without extra tracking code, and our Preview Tool validates click, pageview, custom event, DataLayer event and revenue goals in real time. That removes re-instrumentation, but it doesn't remove the ledger check. Nothing does. > A significant result is only a candidate until it matches the orders. --- If you're running A/B testing with Google Analytics and you're tired of winners that vanish when finance checks the orders, [Otter A/B](https://www.otterab.com) connects to your GA4 property in one click and reports conversion rate, average order value and revenue per visitor side by side for every variant. Validate your goals in real time before launch, test against the GA4 events you already trust, and ship only the changes that pay. --- Canonical page: https://www.otterab.com/blog/ab-testing-with-google-analytics