# What Is a Test of a Hypothesis in A/B Testing _2026-08-26_ You've launched an A/B test on a checkout button, and the early dashboard looks promising. Variant B is converting better than the control, but the test has only just started. Should you ship B, kill it, or wait until the evidence is strong enough to support the decision? That's the practical problem behind **what is a test of a hypothesis**. In A/B testing, hypothesis testing gives you a disciplined way to judge whether an observed difference reflects a genuine change or could plausibly have appeared through random variation. It connects the academic language of null hypotheses, alternative hypotheses, p-values and significance with the everyday CRO question, **“Can we act on this result?”** ## The Decision Every A/B Test Forces You to Make A growth team changes the checkout button from a muted label to a clearer call to action. The control is still running, and the new version is receiving a separate share of visitors. Early results show B ahead of A. The difference looks encouraging, yet the team knows that a small early gap can move in either direction as more visitors arrive. Three options sit on the table: - **Ship B:** Capture a possible improvement quickly, but risk acting on noise. - **Kill B:** Protect the existing experience, but discard a potentially useful change. - **Wait:** Collect more evidence before making an operational decision. Hypothesis testing exists for this tension. It compares the result you observed with the result you'd expect if there were no real difference between the variants. The question isn't whether B looks better in the dashboard. The question is whether the evidence is strong enough to reject the explanation that random allocation produced the gap. > **Practical rule:** An observed lift is a signal to investigate, not permission to deploy. Every experiment should begin with two statements. The **null hypothesis**, written as H₀, describes the default position, usually that the variants perform alike. The **alternative hypothesis**, written as H₁ or Hₐ, describes the effect you're looking for, such as B outperforming A or producing a different result. This preparation matters before traffic is split. UK government guidance recommends a tight, data-driven prediction that connects a change to an expected outcome and a reason, while also warning teams to keep the metric focused so a weak hypothesis doesn't dilute the test's ability to detect a meaningful result. The [UK guidance on running A/B testing Google Ads effectively](https://www.keywordme.io/blog/how-do-i-run-a-b-tests-for-google-ads-effectively) is useful context for marketers translating that discipline into campaign decisions. ![A hand operating a laptop displaying an A/B testing dashboard for comparing two website design variations.](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/01c9823a-e1ec-4930-a20c-87c91adbdd4a/what-is-a-test-of-a-hypothesis-ab-testing.jpg) Historically, the logic has deep roots. John Arbuthnot's **1710 analysis of 82 consecutive years of London birth records** found that male births consistently outnumbered female births. Historical accounts often describe the result as an early formal significance test because such a long imbalance would be extremely unlikely under a 50:50 assumption, helping establish the logic of comparing observed data with a null model ([historical overview of statistical hypothesis testing](https://en.wikipedia.org/wiki/Statistical_hypothesis_test)). ## Null and Alternative Hypotheses Explained Simply A courtroom analogy makes the framework easier to remember. The null hypothesis is like the presumption of innocence. It isn't a statement that the defendant definitely did nothing. It's the starting position. The alternative hypothesis is the prosecution's claim that the evidence supports a different explanation. In an A/B test, the “defendant” is the claim that the variants differ. You begin by treating the apparent difference as explainable by random variation. You then ask whether the collected data would be unusually extreme if that default assumption were true. For the checkout-button experiment, you might write: - **H₀:** The conversion rate for B equals the conversion rate for A. - **H₁:** The conversion rate for B differs from the conversion rate for A. That second statement defines a **two-tailed test**. It treats an increase and a decrease as relevant evidence because the experiment asks whether B is different, not only whether it is better. If your decision was planned specifically around improvement, you could use a one-tailed alternative: - **H₀:** B is no better than A. - **H₁:** B is better than A. The direction must come from the question you wrote before seeing the result. Changing from a two-tailed to a one-tailed test after noticing that B is ahead weakens the integrity of the decision, because the test has been adjusted to fit the outcome. ![A visual explanation of null and alternative hypotheses using the legal analogy of a trial.](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/e556cb4e-0608-4b80-a38c-23ab54c2b251/what-is-a-test-of-a-hypothesis-null-alternative.jpg) ### The null isn't proven true A hypothesis test doesn't prove that H₀ is true. It either finds enough evidence to **reject H₀**, or it **fails to reject H₀**. Those phrases aren't interchangeable. Failing to reject H₀ means the data didn't provide sufficiently strong evidence for the alternative under the chosen testing rules. It doesn't establish that A and B are identical, and it doesn't rule out a useful effect that the experiment wasn't sensitive enough to detect. UK university teaching materials describe hypothesis testing as a decision process involving a test statistic, a significance threshold and a p-value. UK A-level guidance also treats the null hypothesis, alternative hypothesis, significance level, test statistic, one-tailed and two-tailed tests and critical regions as core terms. For a CRO practitioner, the important lesson is simple: **write the question first, then let the evidence answer that question under fixed rules**. ## How a Test Statistic, P-Value, and Significance Work Suppose variant B shows a higher conversion rate than A, and your team must decide whether to ship B or keep the current page. A frequentist z-test turns that observed gap into a comparison against the variation expected from sampling. The basic structure is: `z = (conversion rate for B - conversion rate for A) / standard error` The standard error estimates how much the difference could fluctuate if the null hypothesis were the working model. For a two-variant conversion test, the calculation uses both conversion counts, both sample sizes and a pooled conversion proportion. The result is a **z-statistic**, which measures the distance between the observed difference and the null expectation in standard-error units. A z-statistic of **1.87** places the result 1.87 standard-error units from the null centre. Under a two-tailed normal model, that corresponds to a p-value of roughly **6.2%**. In plain English, a result at least that extreme would occur with that probability if the null hypothesis were true. These figures explain the mechanics only. They do not describe a real campaign. The p-value is not the probability that B will win future traffic, nor the probability that H₀ is true. It measures how compatible the observed result is with the null model. You compare it with alpha, the threshold chosen before the experiment begins. UK teaching commonly presents **5% or 1% significance thresholds** ([Sheffield's hypothesis-testing guidance](https://sheffield.ac.uk/mash/stats-resources/hypothesis)). A 5% threshold corresponds to the familiar **95% confidence threshold** used in many A/B testing workflows. If the p-value is below 0.05, the result is statistically significant under that rule. In an Otter A/B workflow, this is the point where statistical evidence can support a practical ship-it decision, provided the result also fits the test question and business context. ### A quick lookup | z-statistic | Two-tailed p-value | Significant at 95%? | |---|---:|---| | 1.64 | Approximately 10% | No | | 1.87 | Approximately 6.2% | No | | 1.96 | Approximately 5% | Borderline at the conventional threshold | | 2.58 | Approximately 1% | Yes | The table shows why a visually attractive lift can still fall short of the selected threshold. A result may point toward B while providing insufficient evidence to reject H₀. > **Read the decision correctly:** Rejecting H₀ provides evidence against equal performance. Failing to reject H₀ provides insufficient evidence for a difference, not proof that the variants are equivalent. For a plain-English explanation of p-values in experimentation, see [Otter A/B's p-value explanation](https://www.otterab.com/blog/p-value-explanation). Statistical significance belongs alongside standard errors and confidence intervals, which describe the uncertainty around the estimated effect. General background on statistical methods is available in the [Office for National Statistics methodology resources](https://www.ons.gov.uk/methodology/methodologytopicsandstatisticalconcepts). ## Inside the Z-Test That Powers Most A/B Testing Tools The dashboard version of a frequentist z-test starts with four inputs: **conversions in A, visitors in A, conversions in B and visitors in B**. From these values, the system calculates each conversion rate and the difference between them. The pooled proportion is the combined conversion rate used to estimate the null-world variability: `p pooled = (conversions in A + conversions in B) / (visitors in A + visitors in B)` The standard error then estimates how much the observed difference could vary because the test uses samples rather than every possible visitor: `SE = √[p pooled × (1 - p pooled) × (1/nA + 1/nB)]` The z-score divides the observed rate difference by that standard error. A matching p-value expresses how unusual the result would be under H₀. This is the academic chain translated into dashboard fields: **conversion rate, uplift, z-score, p-value and significance status**. ![Screenshot from https://app.otterab.com/results](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/c0bd5878-c7b4-4ac5-9176-a36b7c4e13d4/what-is-a-test-of-a-hypothesis-ab-testing.jpg) ### What the result screen should tell you A meaningful report shouldn't force you to choose between statistical evidence and commercial context. You need to see the observed conversion rates, the estimated uplift, the significance result and the uncertainty around the estimate. A confidence interval is especially important because it presents a plausible range for the underlying effect rather than making a single point estimate look more precise than it is. At the conventional **95% confidence threshold**, a p-value below **0.05** is commonly treated as statistically significant ([UK government guidance on better data decisions](https://insidegovuk.blog.gov.uk/2017/09/06/making-better-decisions-with-data/)). The label is useful, but it isn't a substitute for checking how the estimate was produced or whether the result matters to the business. Otter A/B applies a frequentist z-test to variant comparisons and presents the resulting significance, uplift and conversion-rate columns in its results view. That lets a marketer inspect the visible inputs and outputs without manually reconstructing every calculation. For a more technical treatment of the method, see [Otter A/B's guide to the z-test in statistics](https://www.otterab.com/blog/z-test-in-statistics). The operational interpretation is still yours. A green significance indicator can support a ship decision when the direction, effect size, confidence interval and test design all align. It shouldn't encourage a team to ignore sample planning, external events, audience mix or the cost of implementing the change. ## Pitfalls That Make Hypothesis Tests Mislead You Statistical significance can make a result look finished before the business question is settled. A test may reject H₀ and still produce a change too small to justify engineering work, design debt or a more complicated customer journey. The first check is **practical relevance**. UK experimental guidance from NC3Rs recommends identifying the effect size that matters before an experiment begins, rather than focusing only on effects that are statistically detectable. A tiny improvement can be statistically significant while contributing little commercial value, especially when implementation introduces cost or operational risk. The second check is peeking. Teams often open the dashboard repeatedly, see the first promising result and stop the test immediately. Repeatedly making a ship-or-kill decision at unplanned checkpoints changes the error properties promised by the original threshold. The nominal 5% rule no longer describes the same fixed decision process. ### Four safeguards for a more reliable call - **Check sample size:** A small sample can make conversion rates unstable, so wait for the planned evidence rather than reacting to every early movement. - **Control repeated testing:** Don't turn daily dashboard checks into repeated opportunities to declare significance. - **Protect independence:** Make sure a visitor isn't counted in a way that lets related observations distort the comparison. - **Judge effect size:** Ask whether the estimated change clears the minimum effect worth implementing, not only whether its p-value crosses the threshold. ![An infographic titled Pitfalls That Make Hypothesis Tests Mislead You, featuring four key warnings for statistical analysis.](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/32fba3ff-880a-4641-9745-9cb915a6d9e2/what-is-a-test-of-a-hypothesis-hypothesis-pitfalls.jpg) Confidence intervals add another useful diagnostic. A narrow interval that sits on the beneficial side supports a clearer decision than a wide interval that includes both worthwhile gains and harmful declines. Segment checks can reveal whether the overall result depends on an unusual audience mix, although segment analysis should be planned carefully rather than used to search endlessly for a favourable subgroup. > **Significance is necessary, never sufficient.** ## Sample Size and Power Before You Launch Before launching an A/B test, decide what evidence would justify a ship or kill decision. **Sample size** is the traffic needed for the comparison. **Statistical power** is the test's ability to detect an effect of a chosen size when that effect is real. The **minimum detectable effect**, or MDE, is the smallest improvement worth acting on. Use four inputs to set the plan: | Parameter | Typical value | Role in calculation | |---|---|---| | Baseline conversion rate | Your current rate | Anchors the expected outcome | | Minimum detectable effect | The smallest worthwhile change | Defines the effect the test must detect | | Significance level | 5% | Sets the false-positive threshold | | Statistical power | 80% | Sets the target ability to detect the chosen effect | The **5% significance level** and **80% power** are conventional planning values, not rules for every experiment. A smaller sample gives the test less ability to detect a real improvement, increasing the chance of a Type II error. Choose these settings before launch so the decision rule is not adjusted to fit the result. The trade-off is straightforward. Holding the confidence and power targets steady, detecting a smaller effect generally requires more traffic. Cutting the MDE in half can require roughly doubling the required sample as a mental shortcut, although the exact calculation depends on the baseline rate, allocation and test design. ### A pre-launch checklist 1. **Choose the baseline:** Use the current conversion behaviour for the metric you will test. 2. **Set the MDE:** Define the smallest change that would justify shipping the variant. 3. **Choose alpha and power:** Select the significance threshold and desired detection capability before launch. 4. **Lock the output:** Treat the calculator's visitor requirement as the stopping condition, not a suggestion. 5. **Launch only after planning:** Avoid changing the target halfway through the experiment. A sample-size calculator converts these choices into a traffic requirement. A duration estimate then helps you judge how long collection may take, given the site's normal volume. If the required sample is too large for the available traffic, change the MDE, test design or business question before launching, rather than hoping an underpowered test will settle the decision. Otter A/B's frequentist workflow applies the same logic to a practical 95% confidence threshold. Its [guide to statistical power](https://www.otterab.com/blog/what-is-statistical-power) explains why a variant can appear ahead yet fail to provide enough evidence for a ship decision. Planning power first makes that outcome interpretable: the test may not have shown that the idea is ineffective, only that the planned evidence was insufficient. ## Turning the Result into a Confident Decision A completed test needs a decision protocol, not a celebration of whichever column is green. Start by confirming that the planned sample and test duration were met. If the experiment stopped early, the reported p-value may not support the fixed decision rule the team agreed to use. Then check the statistical evidence: 1. **Confirm the p-value:** Did it cross the preselected significance threshold? 2. **Read the interval:** Does the confidence interval support a beneficial direction, or does it still include zero? 3. **Assess the lift:** Is the estimated effect at least as large as the MDE? 4. **Check business impact:** Would the change improve the target outcome after implementation costs and risks? 5. **Choose the operational status:** Ship, kill, iterate or extend. A significant positive effect that clears the MDE supports shipping, assuming the test was valid and the result fits the broader customer experience. A significant result with trivial magnitude may justify iteration rather than deployment. A wide interval that overlaps zero calls for more evidence, while a flat outcome after the experiment reached sufficient power can support killing the idea. The dashboard should help you place these signals beside one another. The conversion-rate columns show what happened, uplift describes the observed difference, significance shows how the result compares with the null model, and confidence intervals show the uncertainty around the estimate. None of those fields replaces judgement, but together they make the judgement auditable. Record the hypothesis, audience, primary metric, decision rule, result and learning. Feed that learning into the experimentation roadmap, then queue the next test instead of treating one result as a final answer. --- Otter A/B helps teams define an experiment, plan sample size and duration, compare conversion outcomes and validate statistical significance before calling a winner. Visit [Otter A/B](https://www.otterab.com) to connect frequentist hypothesis testing with a practical ship, kill or iterate workflow. --- Canonical page: https://www.otterab.com/blog/what-is-a-test-of-a-hypothesis