# What Sample Size Is Statistically Significant? Guide _2026-10-13_ You've launched an A/B test for a new checkout headline. The dashboard shows a promising difference, one version appears to be ahead, and a stakeholder asks whether the team can roll it out today. The sample looks large enough because the visitor count has moved beyond a familiar benchmark. Two weeks later, the result reverses. The problem wasn't necessarily the test idea. It was the decision rule. **There is no universal sample size that automatically becomes statistically significant.** Reliability depends on the size of the observed difference, the variability in the data, the traffic split, the conversion rate, and the confidence threshold you choose. For UK growth teams using Otter A/B, the more useful question is whether each variant has enough observations to estimate the conversion-rate difference precisely enough to support a decision. ## Why Most A/B Tests Fail Before They Start A product manager sees the first result on Monday morning. Variant B has more completed checkouts than Variant A, so the team pauses the experiment and ships the new design. The uplift looks persuasive in the dashboard, but the test has collected only a thin layer of evidence. A handful of purchases, an uneven traffic split, or a short-lived campaign effect can make the early leader look stronger than it really is. The team celebrates, then watches performance settle back to its previous level. Engineering has already spent time implementing the change, marketing has updated the campaign, and nobody can say whether the original result was a genuine improvement or ordinary sampling noise. ![A stressed woman working on a laptop at a messy desk with A/B testing data analytics.](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/7e4a2ca7-f214-48f9-b3c6-f67682df174c/what-sample-size-is-statistically-significant-ab-testing-stress.jpg) ### The dashboard is not a stopping rule Many teams treat a test like a race towards a magic visitor count. They launch the experiment, refresh the dashboard, and stop as soon as the confidence indicator crosses a preferred line. That process confuses **when a result looks encouraging** with **when the estimate is precise enough to act on**. Sample size matters because every conversion rate is an estimate of an underlying customer behaviour. With fewer observations, random variation has more influence. A change in traffic quality, device mix, weekday pattern, or campaign audience can move the estimate substantially. More observations generally narrow that uncertainty, but they don't guarantee a meaningful or commercially useful result. > **Practical rule:** Don't ask whether the test has reached a universal number of visitors. Ask whether each variant has enough outcome observations to make the uncertainty around the difference acceptably narrow. ### Start with the decision, not the count Before launch, define the smallest improvement worth implementing. A minor conversion-rate change may not cover development, quality assurance, design, analytics, and ongoing maintenance. A larger improvement might justify a shorter test because the business value is easier to establish, although the result still needs enough evidence to avoid a fragile decision. The most reliable teams write down the baseline conversion rate, the commercially meaningful uplift, the traffic allocation, the primary goal, and the stopping rule before they inspect results. That preparation prevents the dashboard from turning an uncertain early signal into a premature winner. ## What Statistical Significance Actually Means Statistical significance is a statement about evidence under a specified model. It doesn't mean that a result is important, guaranteed to repeat, or valuable to the business. It means the observed difference would be relatively unusual if there were no underlying difference between the variants. The UK Office for National Statistics explains confidence intervals through an estimate plus or minus **1.96 standard errors** for a 95% interval. Across repeated samples, approximately **19 out of 20** such intervals would be expected to contain the true population value, while **1 out of 20** would not, as described in the [ONS guidance on uncertainty and measurement](https://www.ons.gov.uk/methodology/methodologytopicsandstatisticalconcepts/uncertaintyandhowwemeasureit). This is a repeated-sampling interpretation, not a promise that one particular interval has a fixed probability of containing the truth. ![An infographic explaining the concept of statistical significance, its meaning, p-values, and common thresholds for researchers.](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/55cd1639-fb82-493f-b98b-5264793e48f2/what-sample-size-is-statistically-significant-statistical-significance.jpg) ### Confidence intervals show the uncertainty Suppose Variant B has a higher observed conversion rate than Variant A. The key question isn't only how far apart the two observed rates are. You also need to know how uncertain that difference is. A wide interval means the data are compatible with several plausible underlying effects. The change might be meaningfully positive, close to no difference, or even negative. A narrow interval gives the product team a clearer estimate and makes a decision less dependent on random variation. The ONS notes that increasing sample size reduces standard errors, which makes confidence intervals narrower and estimates more precise. That explains why sample size is better understood as a **precision problem** than as a fixed gateway to significance. ### The 95% threshold is a convention A two-sided 95% confidence threshold corresponds to a 5% significance level. Under the null hypothesis of no difference, the threshold represents the probability of observing a result at least this extreme by chance. It isn't a law of nature, and it doesn't rescue a poorly designed experiment. Baseline conversion rate, traffic allocation, outcome variance, and the minimum effect worth detecting all affect the evidence produced by a test. A small sample can produce a significant result when the observed difference is large and consistent. A much larger sample may be necessary to detect a small change in conversion rate. A useful mental model is simple. **Significance asks whether the evidence is strong against no difference. Precision asks how tightly you have estimated the difference.** Your rollout decision needs both statistical evidence and an effect size that matters to customers and the business. ## How to Calculate the Right Sample Size A sample-size calculation starts before the test goes live. It doesn't begin with a generic visitor target, because the required observations depend on the question you're asking. ### Define the four inputs 1. **Baseline conversion rate.** Use the current performance of the page and goal you plan to test. A checkout purchase goal has different variability from a button-click goal, so don't substitute a convenient funnel metric just because it produces more observations. 2. **Minimum detectable effect.** Specify the smallest relative or absolute improvement worth detecting. If a Shopify team would only implement a redesign when it produces a commercially meaningful uplift, that threshold should drive the calculation. 3. **Desired power.** Power describes the chance of detecting the chosen effect if that effect genuinely exists. A low-powered test may miss a useful change, while a very large test can detect a difference too small to justify implementation. 4. **Confidence threshold.** Choose the error threshold before reviewing results. Otter A/B uses a frequentist test at a 95% confidence threshold, but the important planning principle is to select the threshold in advance and apply it consistently. ### Work in conversions per variant Traffic is only a proxy for evidence. If the goal is purchase, the meaningful observations are purchases and the conversion-rate uncertainty created by those purchases. Two tests with the same visitor count can have very different precision when their baseline rates, traffic sources, or outcome patterns differ. Enter the four inputs into a sample-size calculator, then record the required observations for each variant. The [Otter A/B sample size calculator](https://www.otterab.com/free-tools/sample-size-calculator) is designed around this planning logic, helping teams estimate the visitors or conversions needed to detect a specified effect at a chosen confidence level and power. A practical planning sequence looks like this: - Record the baseline rate from a comparable period. - Choose the smallest uplift that would justify the implementation cost. - Set the primary goal and avoid changing it after launch without saying so. - Allocate traffic deliberately and account for uneven exposure. - Calculate the required observations per variant. - Add operational time for normal traffic cycles, quality checks, and possible exclusions. - Stop according to the pre-agreed rule, not because one daily reading looks favourable. For teams testing several versions or interacting page elements, it also helps to understand [how FindClout approaches creative testing](https://findclout.com/blog/multivariate-testing). The same discipline applies, define the comparison, the outcome, and the effect worth detecting before interpreting the result. A more complex design can spread observations across combinations, making a raw visitor count even less informative. The output isn't a promise that the test will reach significance. It tells you what the design is capable of detecting with the assumptions you selected. If your store cannot support the required observations, change the experiment design or accept that the result will remain directional rather than definitive. ## Power, Error, and Why Precision Matters More Than Count A high-traffic landing-page test and a low-volume purchase test don't carry the same statistical burden. A page can generate many button clicks while producing relatively few orders. If the team treats every event as equally informative, it may declare success on an intermediate action that doesn't predict commercial value. **Power** protects against false negatives. It reflects the probability of detecting a real effect of the size you planned to detect. A test with weak power can produce a non-significant result even when the treatment helps, so “no winner” shouldn't automatically mean “no effect”. **The error threshold** protects against false positives. If the team checks many goals, segments, variants, and time windows, the opportunity to find an apparently positive result by chance grows. That risk needs to be addressed in the test plan rather than hidden inside repeated dashboard checks. ### Precision depends on design The National Centre for Social Research reported that questions asked across the British Social Attitudes survey generally had a margin of error of approximately **plus or minus 2 to 3 percentage points at the 95% confidence level**, as documented in its [British Social Attitudes technical details](https://natcen.ac.uk/sites/default/files/2025-06/BSA%2042%20Technical%20Details.pdf). The same documentation reported an overall design effect of **1.43**, meaning the sampling design created more uncertainty than an equally sized simple random sample. That example matters because data collection design changes precision. In experimentation, uneven traffic, clustered audiences, repeated exposure, device differences, and funnel dependencies can all affect how much information each observation contributes. A simple visitor total can therefore overstate the strength of the evidence. ### Compare the decision you can support A well-powered high-traffic test may estimate a small effect precisely. A low-volume purchase test may need a longer run, a larger commercially meaningful effect, or a broader funnel event before it can distinguish signal from noise. Neither situation has a universal visitor answer. Use [Otter A/B's guide to statistical power](https://www.otterab.com/blog/what-is-statistical-power) when setting the test assumptions, then report the observed effect alongside its confidence interval. That combination tells stakeholders what happened and how much uncertainty remains, rather than reducing the outcome to a winner or loser label. ## Low-Volume Experiments and Subgroup Testing Low-volume stores often ask the wrong first question: “How many visitors do we need?” Their real problem is usually that the chosen outcome is rare. If purchases arrive slowly, a purchase-based test may take too long to produce a precise comparison, even when the page receives steady traffic. UK public-sector survey practice offers a useful principle. NHS England specifies a minimum of **200 responses** for reliable results in each contract area and illustrates a 50% satisfaction estimate with a **95% confidence level and a plus or minus 5 percentage-point margin of error**, spanning 45% to 55%, in its [NHS111 Patient Survey specification](https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2022/12/NHS111-Patient-Survey-Specification-2022-v1.0.pdf). Those figures don't transfer directly to an online shop, but they show that organisations define reliability through precision rather than a magic universal count. ### Change the experiment when the funnel is sparse A low-volume e-commerce team has several defensible options: - **Choose a broader event.** Test add-to-cart, checkout initiation, or another behaviour that occurs more often, then treat purchase revenue as a secondary validation metric. - **Run for longer.** Extend the observation window across comparable trading conditions instead of stopping after a convenient calendar period. - **Combine comparable periods.** Pool evidence only when the audience, offer, tracking, and experience remain sufficiently comparable. - **Raise the minimum detectable effect.** If the store cannot detect a subtle improvement, plan around a larger change that would justify the decision. - **Reduce unnecessary variants.** Splitting scarce traffic across many versions weakens the evidence available to each comparison. Another UK government survey suppresses subgroup estimates below **30 respondents** because the margins of error are considered too unreliable. For an e-commerce team, that is a warning against treating a tiny device, channel, location, or customer segment as a reliable standalone winner. ### Treat segments as planned questions Every extra segment or goal creates another opportunity for a chance result. Don't browse subgroups until one produces a favourable number, then present it as the main conclusion. Define the primary audience and outcome first, and label exploratory patterns as exploratory. Sequential monitoring needs the same discipline. A team that repeatedly checks significance can alter its false-positive risk unless the analysis accounts for the stopping process. The [Otter A/B guide to sequential testing](https://www.otterab.com/blog/sequential-testing) provides context for monitoring experiments while preserving a defensible decision rule. ## Statistical Versus Commercial Significance A statistically significant result can still be a poor product decision. A large dataset may estimate a tiny improvement with high precision, yet the change may not cover the cost of design, engineering, testing, analytics, maintenance, or operational disruption. Statistical evidence answers whether the observed difference is distinguishable from uncertainty. It doesn't answer whether the difference deserves investment. The reverse also matters. A small test can show a promising effect without enough power to establish it confidently. That result may justify a follow-up experiment, not an immediate permanent rollout. The team should describe it as an encouraging signal and keep the uncertainty visible. ![A chart comparing statistical versus commercial significance, showing that large sample sizes can yield unprofitable results.](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/36b71f84-6be0-4f58-80d4-3eb3baa045c6/what-sample-size-is-statistically-significant-significance-comparison.jpg) ### Use evidence as a continuous measure UK statistical education guidance cautions against treating p-values as a binary “significant” or “not significant” label. The result contains a degree of evidence, and the confidence interval shows the range of effects compatible with the data. Reporting only a badge hides whether the likely effect is tightly estimated or still broad. A product review should include: | Decision input | Question to answer | |---|---| | Estimated effect | How did the variants differ on the primary goal? | | Uncertainty | How wide is the confidence interval around that difference? | | Commercial threshold | Is the plausible effect large enough to justify rollout? | | Business outcome | What happened to purchases, revenue, or average order value? | | Implementation cost | What will design, engineering, QA, and maintenance require? | | Test integrity | Did exposure, tracking, seasonality, or audience quality change? | ### Apply three practical stopping rules **Stop and roll out** when the pre-defined threshold is met, the confidence interval supports an effect that clears the commercial bar, and the primary metric agrees with the relevant business outcomes. **Stop without a rollout** when the evidence is sufficiently precise to rule out an improvement worth pursuing. A non-significant result with a narrow interval around negligible effects can be more informative than a vague result from a small test. **Extend or redesign** when the interval is wide, the test is underpowered, the purchase goal is too rare, or external conditions disrupted the comparison. Extending isn't a licence to wait for a preferred outcome. Set the additional evidence requirement before continuing. > **Decision rule:** Report the effect, its uncertainty, the planned detectable effect, and the commercial consequence together. “Variant B won” is not a complete experiment result. Otter A/B can support this workflow by comparing variants, tracking conversion rates and revenue outcomes, and displaying statistical results at its stated confidence threshold. Use those results as part of a pre-planned decision process, not as a substitute for defining what a worthwhile improvement looks like. --- Set the baseline, minimum worthwhile uplift, primary goal, and stopping rule before launching your next experiment, then use [Otter A/B](https://www.otterab.com) to monitor variant performance and connect conversion results with purchases, average order value, and revenue. Start with a clearly defined decision threshold so your team can act on evidence that is both statistically credible and commercially useful. --- Canonical page: https://www.otterab.com/blog/what-sample-size-is-statistically-significant