# How to Calculate Significant Difference for A/B Tests _2026-10-22_ You're staring at a dashboard on a Monday morning. Your new headline is converting ahead of the control, your manager wants it shipped, and you're not sure whether you're looking at a genuine improvement or a lucky run of orders. That uncertainty is normal, and it's exactly why you need to **calculate significant difference** properly before you act. The maths isn't difficult once you see what each step is doing. The harder part, which many guides skip, is deciding whether a real difference is also large enough to be worth money, time and effort. ## Why the Difference Between Two Conversion Rates Matters A marketer I picture often in teaching is running a headline test on a product page. Variant B looks ahead of the control, the team is pleased, and someone asks why they can't roll it out today. The honest answer is that **two variants will almost never produce identical conversion rates**, even if the headline change does nothing at all. Visitors arrive in a random order. A few extra buyers land in one group early on, a traffic source behaves differently on a Tuesday, and the rates drift apart. That drift is **sampling variation**, and it's present in every experiment you run. A raw gap between rates tells you what happened in your sample. It doesn't tell you what would happen if you showed the winner to everyone. ![A hand-drawn illustration showing a magnifying glass comparing blue and green bar charts labeled Variant A and B.](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/305b41a3-a8ba-4100-95d0-84a1de494dcd/calculate-significant-difference-ab-testing.jpg) ### What You're Really Asking When you calculate significant difference, you are asking one focused question. Could a gap this large plausibly appear by chance if both variants convert at the same rate? If the answer is yes, you don't have evidence for a rollout yet. If the answer is no, you have statistical evidence, though you still need a business judgement. That distinction matters more as programmes grow. Larger organisations running many concurrent tests face the same problem at scale, which is why resources on [enterprise digital experience testing](https://www.kogifi.com/articles/a-b-n-testing) stress disciplined variant design before anyone celebrates an early lead. More variants and more dashboards create more chances to mistake noise for insight. > **Practical rule:** Treat every observed conversion rate as an estimate with uncertainty around it, not as the true rate. Think of each rate as a range rather than a fixed point. With little traffic, that range is wide and the two ranges overlap heavily. With more traffic, the ranges narrow and a genuine gap becomes easier to separate from luck. Significance testing formalises that intuition so your decision doesn't depend on who argues most confidently in a meeting. ## The Core Concepts Behind Statistical Significance Start with the **null hypothesis**. It states that both variants share the same true conversion rate and any gap you've seen is chance. The **alternative hypothesis** states that the true rates differ. Your test doesn't prove the alternative. It asks whether your data are surprising enough, under the null, to reject it. The **p-value** captures that surprise. It's the probability of seeing a difference at least as large as yours if the null were true. A small p-value doesn't mean your variant is certainly better, and it doesn't measure how much better it is. Students often reverse this logic, so keep the definition anchored to the null. ![A diagram outlining core concepts of statistical significance including null hypothesis, alternative hypothesis, p-value, and significance level.](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/714adcd8-5437-4f9a-aaf5-4422fa6f7be3/calculate-significant-difference-statistical-significance.jpg) ### The 5% Threshold and the Interval Around the Difference You compare the p-value with a **significance level** chosen in advance, commonly **5%**. The [ONS methodology for measuring uncertainty](https://www.ons.gov.uk/methodology/methodologytopicsandstatisticalconcepts/uncertaintyandhowwemeasureit) explains that the ONS commonly uses a **5% significance level**, calculates the difference and its **standard error**, then builds a **95% confidence interval** around that difference. If zero is excluded, the difference is statistically significant at the **5%** level. If zero is included, you can't rule out chance variation. That gives you a clear decision rule: - **Interval excludes zero:** You've found a statistically significant difference. - **Interval includes zero:** Your evidence is insufficient, whatever the dashboard headline says. - **Interval is very wide:** Your sample is telling you it doesn't know yet, even if one rate looks ahead. The **standard error** is the engine behind the interval. It measures how much your estimated difference would wobble if you repeated the experiment. Larger samples shrink the standard error, so intervals narrow. Smaller samples leave you with wide intervals that often cross zero, which is the maths politely asking for more data. Retail teams testing journeys across devices and stores face the same logic, and guidance on how to [test omnichannel retail growth](https://www.reddog.group/blogs/unleashing-insights/ab-testing-retail-omnichannel-growth) is useful here because a channel mix can widen uncertainty if you slice results too thinly. Keep your primary comparison simple first. You can explore segments after you've established whether the main difference is real. ## The Z-Test Formula Explained with a Worked Example For two conversion rates, the common tool is a **two-proportion z-test**. The idea is simple. Take the difference between the rates and divide it by the standard error of that difference. You're asking how many standard errors your observed gap sits away from zero. The visual below breaks that journey into four manageable steps. ![A four-step infographic illustrating the process of calculating a Z-test score for conversion rate comparison.](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/126d8ef5-a7e8-45fb-ba02-7e7461dba70b/calculate-significant-difference-z-test-formula.jpg) ### Work Through Real Numbers Step by Step Suppose your control receives **1,000 visitors** and **120 conversions**. Your variant also receives **1,000 visitors** and produces **140 conversions**. Don't rush to celebrate the extra orders. Work through the calculation in order. 1. **Calculate each conversion rate:** Control is 120 divided by 1,000, which is **0.12**. Variant is 140 divided by 1,000, which is **0.14**. 2. **Find the observed difference:** Subtract the control rate from the variant rate. That's **0.14 minus 0.12**, giving **0.02**, or two percentage points. 3. **Calculate the pooled proportion:** Combine all conversions and visitors. That's 260 divided by 2,000, giving **0.13**. Pooling reflects the null hypothesis, which assumes one shared true rate. 4. **Compute the standard error:** Use the square root of pooled proportion multiplied by one minus pooled proportion, multiplied by the sum of one over each sample size. For these numbers, the standard error is about **0.015**. 5. **Divide to get the z-score:** Divide **0.02** by **0.015**. Your **z-score is about 1.33**. Now compare **1.33** with **1.96**, the critical value for **95% confidence** described in the ONS framework covered earlier. Your score falls short, so this difference isn't statistically significant yet. The variant may be better, but your sample hasn't shown it convincingly. The short video below walks through the same logic visually, which helps if formulas on a page feel abstract. ### When a T-Test Fits Better A **t-test** follows the same difference divided by standard error logic. It's typically more suitable for smaller samples or continuous outcomes such as revenue per visitor, where values aren't a simple converted or not converted proportion. The calculation uses a t distribution and degrees of freedom instead of the normal approximation behind the z-test. > Don't choose a test because its name sounds advanced. Choose it because it matches your outcome type and sample. Most testing platforms automate this choice. Your job is to understand what the engine is doing so you can spot a result that doesn't make sense. ## Interpreting Your Results Without Overthinking Them Your z-score places the result in one of three zones. Above **1.96**, the variant shows a statistically significant increase. Below **minus 1.96**, it shows a significant decrease. Between those values, you don't have enough evidence to declare a winner. That's the same decision as checking whether the interval around the difference excludes zero. The diagram below maps those zones onto a single number line. ![A diagram illustrating how to interpret Z-score values to determine significant increases or decreases in data.](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/93dde9cb-d419-4243-b6fe-8fea6998a457/calculate-significant-difference-z-score-interpretation.jpg) ### Reading the Worked Example Honestly Your score of **1.33** sits in the middle zone. A two percentage point lead sounds exciting in a meeting, but with these sample sizes it could easily arise by chance. Continuing the test is the correct response, not a failure. If you want the interval view in more depth, the guide on [how to calculate a confidence interval for A/B tests](https://www.otterab.com/blog/how-to-calculate-confidence-interval) shows how the range around the difference is built. A score very close to zero carries a different message. It suggests the variants performed almost identically in your sample, though it still doesn't prove the true rates are equal. Students often treat not significant as proof of no effect. Resist that shortcut, especially when traffic was low. ### Statistical Significance Is Not Practical Significance A result can clear **1.96** and still not matter commercially. [University of York guidance on sample size and detectable differences](https://www-users.york.ac.uk/~mb55/clust/bupa.htm) states that sample size depends on the difference a study is designed to detect, and that smaller expected differences require larger samples. The same guidance context notes that UK online sales represented **more than 28% of retail sales in 2025**, while online spending rose **5% year on year in the third quarter**, so a large experiment can declare a tiny improvement significant even when it won't cover implementation, discounting, fulfilment or media costs. Ask two questions in sequence. Is the difference real? Then, is it worth acting on? The first belongs to statistics. The second belongs to your business. ## Common Pitfalls That Create False Winners The most common mistake is **peeking**. You check the dashboard daily and stop the moment the result looks significant. Each check is another chance for random wobble to cross your threshold, so your real false positive risk climbs above the level you intended. Decide your stopping rule before launch, then honour it when you're tempted to intervene. ### Too Many Variants and Too Many Metrics **Multiple testing** creates the same problem in a different form. If you compare several variants and judge them on conversions, revenue, average order value and engagement, something will look impressive by luck. [UK Health Knowledge guidance on significance testing and errors](https://www.healthknowledge.org.uk/index.php/e-learning/statistical-methods/practitioners/significance-testing-type1-type11-errors) recommends predefining the significance level, identifies multiple testing as a problem, and gives a Bonferroni example where five independent outcomes would require a **1% threshold rather than 5%**. Use that logic to organise your own experiment before traffic arrives: - **One primary outcome:** Choose the single business metric that decides the test. - **Secondary metrics as diagnostics:** Use them to understand behaviour, not to rescue a losing variant. - **Predeclared threshold:** Write down your significance level and stopping rule where the team can see them. - **Planned sample size:** Use [guidance on calculating sample size before you launch](https://www.otterab.com/blog/how-to-calculate-sample-size) so the test isn't doomed to stay inconclusive. ### A Non-Significant Result Is Not Proof of Equality The UK Health Knowledge guidance above also notes that **power is commonly set at 80% or 90%**, so a non-significant result can reflect insufficient traffic rather than proof that variants are equal. Power is your chance of detecting a real effect of the size you care about. Low traffic, a small true effect, or both can leave you underpowered. > A quiet result often means your experiment couldn't hear the signal. It doesn't mean there was no signal to hear. Treat these pitfalls as governance, not arithmetic. Your formula can be perfect while your process manufactures false winners. ## Combining Significance with Business Impact Before your next test launches, define a **minimum practically important effect**. That's the smallest uplift that would justify building, maintaining and possibly discounting around the winning variant. You set it in pounds, using baseline conversion, traffic volume, average order value and implementation cost, not by staring at a p-value after the data arrive. A headline might raise conversion slightly while attracting lower value baskets. In that case, your conversion winner and revenue winner can be different variants, which is why a single rate shouldn't make the final call alone. An [online revenue impact calculator for experiment results](https://www.otterab.com/free-tools/revenue-impact-calculator) helps you translate an uplift into expected incremental revenue before you commit engineering time. ### From Significance to Decision | Metric | Example Value | Why It Matters | |---|---|---| | **Control conversion rate** | **0.12** from the worked example | Sets the baseline every uplift is measured against | | **Variant conversion rate** | **0.14** from the worked example | Shows the observed gain before significance is checked | | **Z-score** | **1.33** against a **1.96** threshold | Tells you the evidence isn't strong enough yet | | **Minimum worthwhile uplift** | Set by your team before launch | Stops a trivial but significant gain triggering a rollout | | **Expected incremental revenue** | Uplift applied to traffic and order value | Connects the statistical result to money | | **Implementation cost** | Build, fulfilment and media effort | Shows whether the gain survives real world costs | Your decision rule now has two gates. The result must be statistically significant, and the expected gain must clear your predeclared worthwhile threshold after costs. A variant that passes only the first gate is an interesting finding, not automatically a rollout. ## A Practical Checklist for Your Next Experiment Keep this sequence somewhere your team will see it before the next test goes live. - **Define one primary outcome:** Pick the business metric that decides the experiment. - **Set your thresholds first:** Record your significance level, power and minimum worthwhile uplift. - **Plan the sample:** Work out how much traffic you need and commit to a stopping rule. - **Run without peeking for a decision:** Monitor for technical faults, not for an excuse to stop early. - **Calculate the difference properly:** Find the z-score and check whether the **95% confidence interval** excludes zero. - **Apply the money test:** Compare expected incremental revenue with implementation and running costs. - **Record the decision:** Note what you shipped, kept testing or rejected, and why. If you'd like a broader refresher on designing the test itself, the overview of [how to run winning split tests](https://skup.net/blog/what-is-split-testing/) covers variant setup that pairs well with this checklist. Teams that want the arithmetic handled automatically can use **Otter A/B**, whose frequentist z-test engine continuously calculates significance at a **95% confidence** threshold, while you focus on the commercial judgement only a human can make. --- Put this framework to work on your next headline, call to action or layout test with [Otter A/B](https://www.otterab.com), which calculates significance continuously and tracks revenue alongside conversion so you can judge both statistical evidence and business impact. Start your first experiment, set your minimum worthwhile uplift before you launch, and ship only the winners that are real and worth having. --- Canonical page: https://www.otterab.com/blog/calculate-significant-difference