# A B Testing Hypothesis _2026-10-19_ Most A/B testing advice starts with a tidy sentence template. That's not the hard part. The hard part is making sure the idea deserves traffic, the metric reflects the intended outcome, and the team knows in advance what result would change its mind. “Change the button colour to increase conversions” isn't a hypothesis. It's a design request with an outcome attached. A useful **A/B testing hypothesis** is an experiment brief: it names the audience and change, predicts a measurable outcome, explains why the change could work, and sets out how the team will judge the result. ## Why Most A/B Hypotheses Never Earn Their Traffic A backlog full of “try a new CTA” and “make the page cleaner” doesn't tell a team what it expects users to do differently. Those ideas can still be worth investigating, but they aren't ready for an experiment. Without an observed problem, a defined audience, and a measurable prediction, the team is testing a preference rather than a reasoned change. Compare “make the button more prominent” with “For mobile visitors who reach the product page, replacing the text-only add-to-cart link with a fixed, benefit-led button will increase completed purchases per eligible session, because session review shows shoppers scrolling past the current link.” The second claim could be wrong. It also gives the team a concrete variant to build, an outcome to measure, and a reason to run the test. If the result is flat, the team can learn that the proposed change didn't address the observed friction, rather than debating whether the colour was attractive. ### Traffic is not a substitute for discipline A large audience doesn't rescue a vague question. In a GOV.UK organ-donation trial conducted with NHS Blood and Transplant and other public-sector partners, **more than 1 million people** saw one of eight webpage variants over five weeks, and each variant was seen by **more than 135,000 people**. The trial used random assignment and measured registration, giving researchers a structured way to compare alternative messages with the existing design. [The GOV.UK organ-donation experiment](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/267100/Applying_Behavioural_Insights_to_Organ_Donation.pdf) shows that hypothesis discipline matters at scale: the result depends on who sees each version, what outcome is defined, and whether the sample can distinguish an effect from noise. A smaller business needs the same logic, even if its experiment takes longer or can only address a larger expected effect. Random allocation helps make the comparison fair. A clear primary outcome stops the team from deciding after the fact that some other metric was the goal. If you're building hypotheses around customer journeys in hospitality, [Purple Engage's hotel marketing insights](https://engage.purple.ai/en-us/blog/digital-marketing-for-hotels) offer useful context for thinking about audience intent and the moments where a booking experience may create friction. ### Give ideas a right to run Before an idea gets into the testing queue, ask whether a result could disprove it. If the team can call any outcome a success, the hypothesis isn't sharp enough. Review your most recent experiments against that standard: each should specify a meaningful change, a target audience, a directional metric, and a reason grounded in evidence. ## The If/Then/Because Anatomy of a Testable Hypothesis The familiar **if/then/because** structure is useful, as long as each clause does real work. “If we change the page, then conversions will rise, because it will be better” has the form but not the substance. Name the experience being changed, the metric that will move, and the user evidence or behavioural explanation behind the prediction. ![A diagram illustrating a testable hypothesis structure using IF, THEN, and BECAUSE statements with example text.](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/ff5b90f5-e32f-4be1-96e9-90017987c696/a-b-testing-hypothesis-hypothesis-framework.jpg) ### Make each clause falsifiable **If** defines the treatment, not the team's hope. “If we replace the generic headline on the pricing page with a headline that names the main customer outcome…” gives the designer a specific job. “If we improve the page…” doesn't. **Then** specifies the primary metric and direction. “Then more eligible visitors will start a plan” is more useful than “then engagement will improve”. “Engagement” could mean clicks, time on page, or scroll depth, and those measures don't necessarily indicate commercial value. **Because** states the reason the team expects a change. It might come from user interviews, support tickets, funnel behaviour, or an observed usability issue. It isn't a place for a slogan such as “people like simple pages”. If the evidence is thin, say so and treat the test as a learning opportunity, not a high-confidence bet. Three examples show how that changes the work: - **Headline:** If first-time visitors see a headline that describes the outcome of the product rather than its category, then qualified trial starts per eligible landing-page session will increase, because interviews suggest visitors don't understand the benefit from the current wording. - **Pricing layout:** If the recommended plan is presented first with a concise comparison of included features, then plan selections will increase, because users have been asking support to clarify differences between plans. - **Onboarding tooltip:** If new users see a contextual tooltip beside the first setup action, then users completing that action will increase, because product feedback indicates that people miss the next step. A practical audit is simple: **Does the hypothesis name the audience, the exact change, one primary metric, a predicted direction, and a reason supported by evidence?** If not, revise it before building the variant. Resources such as [A/B testing examples for e-commerce](https://www.cartboss.io/blog/what-is-split-testing/) can help teams generate ideas, but each example still needs to be rewritten for its own audience and customer journey. For a working team, the hypothesis should live in the experiment record alongside the analysis plan. GOV.UK guidance recommends a workflow covering research, hypothesis formation, test design, build and quality assurance, live execution, and analysis. It also calls for teams to define the primary outcome, minimum detectable effect, significance threshold, power, exclusions, and stopping rule before launch. [GOV.UK's experimental analysis guidance](https://docs.data-community.publishing.service.gov.uk/analysis/abmv/) makes the point plainly: writing the sentence is only the start of the brief. ## Mapping Your Hypothesis to the Right Metric “More conversions” sounds clear until the team has to choose what counts. A click can rise while purchases fall; conversion rate can stay flat while the value of each visitor changes. The metric map connects the change to the business outcome and makes it harder to claim a win on whichever number happens to look best. ### Set the hierarchy before build Start with one **primary metric** that directly tests the hypothesis. For a benefit-led product-page CTA, that might be completed purchases per eligible session, not button clicks. Then choose a **counter-metric** that could expose an unwanted trade-off, such as completed orders if the primary outcome is add-to-cart actions. Add guardrails for outcomes the team can't afford to damage. For ecommerce, that could include average order value, revenue per visitor, refunds, or checkout completion. Keep the list relevant: a guardrail should protect a real business or customer outcome, not create a fishing expedition for significance. If profitability is the question behind a paid-acquisition experiment, understanding [what POAS means and how it differs from ROAS](https://www.productmetrics.io/blog/poas-vs-roas) can help clarify which business outcome the experiment should ultimately serve. The result of GOV.UK's topic-browsing test illustrates why teams should report changes on both scales. Click-through rose from **61.6% to 65.1%**, an absolute gain of **3.5 percentage points** and roughly a **5.7% relative increase**. The figures describe the same experiment, but answer different questions: how many percentage points changed, and how large was that change relative to the original rate? [GOV.UK's account of the redesigned topic-browsing experience](https://insidegovuk.blog.gov.uk/2022/08/08/rolling-out-a-new-design-for-browsing-gov-uk-by-topic/?ref=vickyteinaki.com) is a useful reminder to choose the scale that makes the decision intelligible. > **Practical rule:** Decide what counts as success before the variant goes live. Don't promote a secondary metric to the primary one because it moved in the direction you wanted. ### Keep the map with the experiment | Component | What it defines | Where it lives in Otter A/B | |---|---|---| | Hypothesis | Audience, change, prediction, and rationale | Experiment record | | Primary metric | The main outcome used to evaluate the prediction | Goal configuration | | Counter-metric | A closely related outcome that could contradict the primary result | Analysis plan | | Guardrails | Business or user outcomes that shouldn't materially worsen | Analysis plan | | Stopping rule | Conditions for ending and analysing the test | Experiment record | Use the same discipline for secondary outcomes. The [case for looking beyond conversion rate](https://www.otterab.com/blog/its-not-all-about-conversion-rate-secondary-metrics-matter) is practical: a result only makes sense when the team can see whether the change improved the intended action without harming what comes next. ## Prioritising Hypotheses with ICE, PXL and RICE A testable backlog still needs a decision rule. Without one, the loudest opinion or easiest design change tends to win the next slot. **ICE, PXL, and RICE** can bring structure to the conversation, but none turns weak evidence into a good experiment. Choose a framework that matches the decision the team needs to make. ![A diagram comparing three popular prioritization frameworks for product development: ICE, PXL, and RICE.](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/cb369306-3838-4373-9fb4-64db7aa1e1a7/a-b-testing-hypothesis-prioritization-frameworks.jpg) ### Pick the lens that fits the decision **ICE**, Impact, Confidence, and Ease, is useful when a small team needs a fast way to compare opportunities. It tends to favour changes that look achievable and likely to move the metric. The trade-off is that teams can overrate “ease” and call a convenient idea a priority without enough evidence. **PXL** is designed to challenge assumptions about a test's potential. Rather than relying only on someone's impact estimate, the team evaluates observable properties and evidence behind the change. That makes it harder for a clever-sounding idea to score well on confidence without support. **RICE**, Reach, Impact, Confidence, and Effort, is helpful when a product or finance stakeholder needs to understand who the change affects and what work it requires. It brings reach and effort into the discussion, but a precise-looking score is still only as credible as the inputs behind it. For early-stage stores, keep the scoring lightweight and prioritise evidence and feasibility. More mature programmes can use a fuller framework when they need to compare competing work across teams. Consider a Shopify fashion brand weighing four ideas: clarify delivery details on a product page, change the homepage hero, add a cart-drawer shipping message, or improve the mobile add-to-cart experience. ICE may favour the delivery details if the team can ship it quickly and has strong evidence of customer confusion. PXL may push the hero down the list if the proposal is mostly a creative hunch. RICE may favour a change on a high-reach journey, even if implementation takes more coordination. The ranking should follow the evidence and the business constraint, not an invented score that pretends to be objective. GOV.UK's browsing test also helps explain why impact estimates need careful reporting: a relative lift can sound large while the underlying absolute movement is more modest. [The prioritisation calculator from Otter A/B](https://www.otterab.com/free-tools/test-prioritization-calculator) can provide a practical place to organise competing ideas, but the team still needs to challenge the assumptions behind each score. A score helps decide what to discuss next. It doesn't prove that a test will win, and it shouldn't override a broken metric, weak tracking, or an unclear hypothesis. ## Example Hypotheses for E-commerce and Onboarding Examples are starting points, not plug-and-play prescriptions. Each needs to match a real friction point on the site, a defined audience, and the event tracking the team can trust. If a team can't explain why the user might behave differently, the idea needs more research before it needs a variant. ### Ecommerce tests **Product detail CTA:** If shoppers viewing a product page see a benefit-led add-to-cart CTA instead of generic button copy, then completed purchases per eligible session will increase, because product feedback shows visitors aren't connecting the item's features with its use. Track add-to-cart as a diagnostic, with completed purchases as the primary outcome; monitor order value as a guardrail. **Cart-drawer shipping message:** If cart visitors see clear shipping-threshold information beside their basket total, then completed purchases per eligible cart session will increase, because shoppers may be uncertain about the delivery cost before checkout. Watch for a rise in cart actions that doesn't carry through to completed orders. **Sticky mobile add-to-cart bar:** If mobile product-page visitors can add the item to their basket without scrolling back to the main CTA, then purchases per eligible mobile session will increase, because the current purchase action becomes harder to reach while browsing product details. QA the bar on small screens and check that it doesn't obscure product options or other essential controls. ![Screenshot from https://www.otterab.com](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/screenshots/b23aec74-ac18-4cf4-a74e-fa48a4e3f4f9/a-b-testing-hypothesis-testing-software.jpg) ### Onboarding tests **First-run checklist:** If new SaaS users see a short checklist of the next setup actions, then completion of the first key activation event will increase, because user feedback suggests that new accounts don't know where to begin. Measure that activation event, not checklist clicks alone. **Value-first email capture:** If visitors to a Webflow landing page see the product's practical value before being asked to submit an email address, then qualified sign-ups per eligible session will increase, because the page gives people a reason to exchange their details. Track downstream qualification as a counter-metric so a rise in low-intent sign-ups isn't mistaken for progress. **Contextual checkout tooltip:** If WooCommerce shoppers see a concise tooltip beside a confusing checkout field, then successful checkout completion will increase, because customer questions point to uncertainty about what the field requires. Make sure the tooltip works with keyboard navigation and doesn't block form controls. Before launch, use the available traffic and baseline outcome rate to estimate whether the experiment can detect a meaningful effect. GOV.UK guidance ties sample requirements to baseline rate, expected effect, significance level, and power, so avoid promising a fixed sample or duration without those inputs. The examples in [Otter A/B's collection of A/B testing examples](https://www.otterab.com/blog/ab-testing-examples) can help spark test ideas, but rewrite each one for your own users and funnel. ## Pitfalls That Break Even Good Hypotheses A sound hypothesis can still produce a bad decision if the team keeps checking results, changes the goal midway, or treats an early dashboard lead as a winner. Repeatedly checking for significance and stopping when a result first crosses a threshold can raise false-positive risk. Testing many variants or secondary metrics creates more chances to find an apparently positive result by accident. The control and variant should run during the same period with users randomly assigned, so external changes affect both groups similarly. Before launch, QA variant rendering, event tracking, allocation, exclusions, and the primary metric. Record the analysis plan in advance, then keep secondary outcomes and unplanned audience slices exploratory unless the experiment was designed to test them. GOV.UK's journey work reported a **30% increase in mobile clicks** and a **14% increase on desktop**, with an estimate of roughly **15,000 additional weekly sessions** reaching the correct-answer page. Those relative lifts are a reason to report the underlying outcome and its business meaning, not just the headline percentage. [The GOV.UK account of improving common user journeys](https://insidegovuk.blog.gov.uk/2017/11/14/using-ab-testing-to-measurably-improve-common-user-journeys/) also shows why the decision metric should be chosen before the test. > **Before launch:** Confirm the hypothesis, primary metric, guardrails, sample-size plan, stopping rule, tracking, and randomisation. If one of these changes after results appear, document why and treat the affected analysis with caution. For common outcomes, keep the decision rule direct: - **Flat result:** Don't force a winner. Check data quality and whether the experiment could detect the effect the team cared about. - **Early winner:** Don't stop because the dashboard looks favourable. Follow the pre-agreed stopping rule. - **Device results disagree:** Treat the split as exploratory unless device-level analysis was planned and adequately powered. --- Otter A/B helps teams run website experiments by setting variants, traffic allocation, conversion goals, and decision criteria in a dashboard. If your hypothesis is ready to test, visit [Otter A/B](https://www.otterab.com) to organise the experiment and evaluate its results. --- Canonical page: https://www.otterab.com/blog/a-b-testing-hypothesis