AOV Testing Requirements: A Practical Guide for 2026

Your bundle test is showing a higher average basket, and the team wants to roll it out. Then someone checks completed orders, another asks about discounts, and finance points out that returns haven't landed yet. Suddenly, the result is less convincing than the dashboard made it look.

Good AOV testing requirements prevent that conversation from happening after the decision. They establish what counts as revenue, who belongs in the analysis, how much evidence you need, and which commercial trade-offs you won't accept.

Why AOV Alone Is a Risky Success Metric

Consider a hypothetical Shopify store testing a larger bundle against individual products. Customers who buy the bundle spend more per order, so the variant's AOV rises. But customers who wanted a smaller purchase leave without buying, while some bundle purchasers later return unwanted items.

The merchandising team sees a stronger basket. Finance sees fewer customers completing purchases and more revenue exposed to refunds. Neither team has necessarily calculated anything incorrectly, they're judging different outcomes.

The purchaser filter changes the question

AOV is a conditional metric: it describes spending among people who placed orders. It doesn't describe the commercial performance of everyone exposed to the experiment. When a treatment changes who purchases, comparing purchaser averages also compares differently selected groups.

That makes AOV useful for diagnosing basket behaviour, but risky as the sole launch criterion. A more expensive basket isn't automatically a more valuable visitor.

The stakes aren't confined to merchandising. The ONS analysis of UK consumer card spending reported that online spending represented 50.5% of total UK card spending in September 2025, compared with 43.7% in September 2019. That's card spending, not a universal retail AOV benchmark, but it reinforces why online experiments need monetary outcomes rather than conversion labels alone.

Separate the basket diagnosis from the business decision

For the hypothetical bundle, the useful questions are specific. Did customers add complementary products, or did the offer discourage smaller purchases? Did the discount change the product mix towards items with weaker margins?

Before changing another offer, use a funnel investigation to locate the friction. Refport's resource to diagnose funnel bottlenecks is a relevant starting point when the uncertainty concerns where shoppers stop progressing.

Practical rule: Treat a higher AOV as an explanation to investigate, not permission to launch.

A defensible decision framework puts revenue per eligible visitor beside AOV, then checks contribution margin and post-purchase outcomes. Keep the measures separate rather than combining them into an opaque score. Stakeholders should be able to see whether the treatment improves basket composition, customer completion, or actual commercial value, and where those outcomes conflict.

Defining the AOV and Revenue Metrics the Test Will Judge

Write the metric contract before configuring the experiment. Your analyst and finance lead should agree on the revenue numerator, order status, eligible population, attribution window and refund treatment. If those definitions change after the result appears, the test becomes difficult to defend.

Start with AOV = total realised revenue ÷ completed orders. Maintain a separately labelled gross view when it helps explain discounting and returns, rather than allowing the dashboard's default revenue field to decide what “realised” means.

Make the accounting choices explicit

The ECDB UK transaction benchmark distinguishes gross AOV from net AOV after discounts and returns. For your experiment, make that distinction operational: define the exact ledger fields behind each view and when an order becomes eligible.

Otter's explanation of what average order value measures can help align terminology before you document store-specific rules. The important step is translating the definition into fields your implementation can collect.

Primary metric options for an AOV test

Metric	Numerator	Denominator	When to use
Net AOV	Realised order revenue after agreed adjustments	Completed orders	Diagnose basket value among purchasers
Net revenue per eligible visitor	Realised revenue attributed to assigned visitors	All eligible assigned visitors	Judge the commercial effect across the exposed population
Contribution margin per visitor	Revenue less agreed variable costs	All eligible assigned visitors	Judge whether additional sales retain useful margin

For most basket experiments, I'd choose net revenue per eligible visitor as the primary outcome and keep net AOV as a diagnostic. Contribution margin may be the stronger business outcome when discounts or fulfilment costs are central to the hypothesis, provided the cost data is dependable.

Document these decisions explicitly:

VAT and shipping: Specify whether each is included, and use the same treatment across variants.
Discounts and cancellations: Distinguish product discounts from order discounts and define cancelled-order handling.
Returns: Agree on the observation window and how partial refunds reduce realised revenue.
Currency: Restrict eligibility to GBP orders or use a documented conversion policy consistently.

Match allocation to the customer journey

Use session-level assignment for a session-scoped intervention, with allocation fixed during that session. Prefer persistent customer-level assignment when repeat visits or logged-in purchases are part of the intended experience. Don't switch denominators between visitors, sessions and orders.

Analyse assigned eligible visitors on an intention-to-treat basis, retaining non-purchasers in the visitor denominator. Also define how subsequent purchases are attributed, rather than crediting whichever variant happened to appear last.

For merchants spanning storefront and marketplace operations, Online Brand Growth's resource to scale Amazon sales with margin provides a relevant companion perspective. Keep marketplace accounting separate, but bring the same margin discipline to your storefront metric contract.

Instrumentation and Data Quality Checks Before Launch

A purchase total can look plausible while being wrong. A confirmation-page refresh may duplicate a transaction, a refund may never reach analytics, or an order value may include VAT in one system but exclude it in another.

For Shopify and WooCommerce experiments, make the order record your reconciliation reference. Browser events establish exposure and behaviour, but the financial readout needs a dependable connection between variant assignment and completed transactions.

Build a QA ticket around failure modes

Use explicit acceptance checks rather than a vague instruction to “verify tracking”. The IRP UK ecommerce measurement workflow highlights test orders, duplicate-event checks, currency and VAT rules, cancelled-order handling, and reconciliation between analytics and payment data.

Event taxonomy: Define what purchase, refund and checkout_complete mean. Don't count a checkout-interface event as settled revenue.
Transaction identity: Attach a stable order identifier and confirm that confirmation-page reloads or event retries don't create extra purchases.
Variant linkage: Persist the assigned variant through checkout and verify that the order can be matched back to the eligible visitor.
Accounting consistency: Compare currency, discounts, VAT, shipping and cancellation status against the agreed metric contract.
Refund updates: Test partial and full refunds, including adjustments arriving after the original purchase event.
Allocation integrity: Inspect observed traffic against the planned split and investigate unexplained discrepancies before interpreting outcomes.

Reconcile transactions, not just totals

Run QA orders through the actual payment and refund paths. Include a discounted basket, an interrupted checkout and a returning customer, then inspect the resulting records rather than trusting a successful event preview.

An aggregate match isn't enough: a missing order and a duplicated order can conceal each other. Reconcile identifiers and amounts, and assign someone responsibility for investigating unmatched transactions. Use a separate reconciliation routine for late-arriving refunds and accounting adjustments.

Otter's guide to data cleaning best practices is useful for formalising those checks. Write exclusion rules before the outcome is visible, particularly for test orders and clearly invalid records.

A soak test should run until the relevant event paths and reconciliation checks have completed successfully. Don't use an arbitrary elapsed-time threshold as proof that refunds, repeat visits or asynchronous payment updates work. Where cross-device identity can't be resolved reliably, document that limitation instead of pretending every customer has been deduplicated.

Sample Size, Power, and Stopping Rules for Revenue Tests

Revenue planning starts with your store's distribution, not an industry AOV. A handful of expensive baskets can move the mean substantially, so a conversion sample-size calculator isn't automatically suitable for a monetary outcome.

The IRP ecommerce benchmark for August reported AOV of £129.23 in August 2026, versus £126.46 in August 2025, a change of £2.77, or 2.19%. The same reference supports planning with 80% power and a two-sided alpha of 0.05. Use the benchmark as context for modest effects, not as your store's expected uplift.

Estimate variance at the level you randomise

If the primary metric is revenue per eligible visitor, assemble historical visitor-level revenue, including non-purchasers. Estimate variability from that distribution, not just the spread of completed baskets.

In a fixed-horizon comparison, the required sample grows with revenue variance and shrinks as the minimum detectable effect becomes larger. That relationship explains why a small commercially meaningful uplift can require substantial exposure. Your current purchase rate and basket variability both affect the visitor-level distribution.

Record the inputs that make the estimate reproducible:

Baseline population: Use eligible traffic with a mix representative of the planned experiment.
Minimum detectable effect: Set the smallest absolute revenue improvement worth acting on.
Statistical specification: Document the power target, significance threshold and analysis method.
Exposure plan: State traffic allocation, required sample and calendar coverage.

Use a revenue-appropriate calculation or simulation, and inspect how unusually large orders affect the estimate. Otter's A/B testing sample-size guide can support the planning conversation, but the inputs still need to come from your merchant data.

Choose the stopping method before exposure

A fixed-horizon test has a planned readout after its sample and calendar requirements are met. Operational monitoring remains necessary, but repeatedly checking significance and stopping at a favourable moment isn't the same analysis.

A significance notification is a monitoring event, not a substitute for the stopping rule.

If stakeholders require early efficacy decisions, use a valid sequential method designed for repeated examination. Separately predefine safety stops and who can return traffic to control when an operational guardrail fails.

Netco Design LLC's resource to compare conversion testing strategies can help structure that discussion. For revenue, also plan a bootstrap confidence interval or sensitivity analysis, and report the median as descriptive context rather than replacing the mean after seeing an inconvenient result.

Segmentation Without Data Dredging

Device-level results can explain an experiment without justifying a device-level launch. The distinction matters when a mobile upsell appears promising but the desktop experience performs poorly.

Consider a hypothetical WooCommerce test where mobile visitors see a compact add-on panel and desktop visitors see a larger product grid. Opposing results could reflect presentation, traffic composition or random variation. A favourable mobile estimate alone doesn't establish which explanation is correct.

Compare planned questions with exploratory findings

Predefine segments with a reason tied to the hypothesis. Device, acquisition channel, new versus returning customers, product category and geography are useful candidates, but they shouldn't all become independent opportunities to declare a winner.

Reporting habit	What it supports	Decision boundary
Pre-specified segment comparison	Evaluating an expected difference	Use the planned uncertainty and multiplicity treatment
Exploratory slicing after readout	Generating a follow-up hypothesis	Don't treat discovery as confirmation
Stratified randomisation	Balancing an important traffic characteristic	Choose strata before allocation
Pre-specified covariate adjustment	Accounting for baseline differences	Don't select adjustments because they favour the treatment

For the mobile-versus-desktop example, inspect the treatment-by-device difference directly. A favourable estimate in one cohort and an unfavourable estimate in another doesn't, by itself, prove that the treatment effects differ.

Set a boundary for cohort-specific decisions

If subgroup conclusions can trigger launches, specify how you'll address multiple comparisons, using Bonferroni or another appropriate correction. Keep the planned family of comparisons narrow enough to remain useful. More slicing isn't a substitute for adequate evidence.

Split deliberately: Consider a cohort-specific launch when the mechanism is plausible, the comparison was planned and the cohort evidence is sufficiently precise.
Investigate cautiously: If the pattern emerged after broad exploration, use it to design a targeted follow-up.
Preserve the primary decision: Don't overturn an unfavourable overall result by searching for a profitable-looking fragment.

Also separate pre-exposure segments from behaviours the treatment may change. Baseline customer type is different from achieved basket size or selected delivery method. Those latter breakdowns can help explain mechanisms, but conditioning on them requires caution when making causal claims.

Guardrail Metrics That Stop You Shipping a Harmful Win

A free-delivery threshold is a pricing and fulfilment experiment, not merely a basket experiment. Customers may add products to qualify, but the merchant still has to fund the shipment and retain enough margin after returns.

UK fulfilment expectations deserve particular attention. DHL's UK ecommerce trends report reports that 80% of shoppers say they abandon when their preferred delivery option is unavailable, while 75% abandon when returns don't meet expectations. Those responses aren't predicted effects for your test, but they justify treating delivery and returns as commercial safeguards.

Set tolerances with finance and operations

Define acceptable deterioration before launch. Use your own cost structure, baseline variability and operational constraints, rather than borrowing generic thresholds that may be inappropriate for your catalogue.

Contribution margin per visitor: Include agreed product costs, discounts, payment costs and variable fulfilment expenses.
Refund and cancellation rates: Specify the denominator and allow enough follow-up for the relevant outcomes.
Checkout completion: Watch for offers that add friction or make the final delivery price harder to understand.
Fulfilment cost: Capture changes in shipment size, split deliveries and other treatment-linked costs.
Delivery availability: Check whether shoppers can still access the options the control experience supports.

Some guardrails should trigger an immediate operational pause, such as broken checkout or missing delivery options. Others need a pre-specified statistical tolerance and adequate observations. Don't confuse an absence of significant harm with evidence that a poorly measured guardrail is safe.

Commercial veto: A larger basket doesn't compensate for unacceptable unit economics.

Make the launch decision a set of gates

Keep the decision framework transparent. Require credible improvement in the primary revenue outcome, then acceptable margin and customer-completion guardrails. Distinguish a clear failure from uncertainty that needs more evidence.

For a bundle, that could mean declining to launch despite a significant AOV increase because realised margin deteriorates. For a delivery threshold, it could mean holding the decision until fulfilment costs and returns mature. These are decision examples, not reported merchant results.

If revenue improves but a guardrail remains unresolved, record the outcome as promising but not launch-ready. Engineering can retain the implementation while finance and operations complete the evaluation. That's a more useful conclusion than forcing every experiment into winner or loser.

Reporting the Result and Turning It Into a Decision

Report absolute uplift in pounds, relative uplift, confidence interval, sample size and exposure duration, using the measurement workflow established earlier. Include number needed to treat only for a clearly defined binary outcome; it isn't a natural summary of continuous revenue.

Give stakeholders a compact decision record:

Evidence: Primary outcome, guardrail status and planned segment findings.
Context: Promotions, stock-outs, shipping changes and calendar coverage.
Action: Launch, reject or continue under the pre-agreed rule.

Use consistent, brandable charts with visible denominators. Revalidate a winner after materially different promotional conditions rather than assuming it transfers unchanged. Hand engineering the approved variant, measurement requirements, rollout owner and rollback conditions, not just a screenshot of significance.

For a lightweight option alongside your analytics and payment records, visit Otter A/B to test website variants and track purchases, AOV and revenue by variant. Start with a documented revenue goal and stopping rule, then use its brandable reports to share the result alongside your reconciled margin and refund guardrails.