# AB Testing Calculator for Sample and Power

_2026-11-12_

A large and growing share of British retail now happens online. When a headline change shifts conversion by two-tenths of a percentage point, that difference is not a rounding error.

An A/B testing calculator helps you judge whether your traffic can measure a change of that size. It connects three planning counts: visitors needed per variant, the conversions those visitors might produce, and the calendar time required to reach the sample. A revenue check and a UK consent check complete the picture.

Treat the calculator as a planning tool, not a winner button. A result can look convincing while the test is still too small, the revenue impact is weak, or consented traffic has not had enough time to accumulate.

## Sample Size for a Conversion Test

At a **2.0% baseline**, detecting a lift to **2.2%** needs roughly **80,000 visitors per variant**, so **160,000 visitors** in a straight 50/50 split, at 80% power and a 5% significance level, as the [UK ecommerce conversion rate planning guide from Solution25](https://solution25.com/uk/ecommerce-conversion-rate-optimization/) sets out. The effect being hunted is a fifth of a percentage point. Teams often arrive expecting a few thousand visitors and leave with a spreadsheet.

Here is the same maths at the baselines UK sites typically run. Figures are visitors per variant for a two-sided test, equal groups, one comparison, a binary conversion.

| Baseline | Relative lift | Visitors per variant |
| --- | --- | --- |
| 2.0% to 2.2% | 10% | about 80,000 |
| 1.68% (UK average) to 1.85% | 10% | about 96,000 |
| 1.68% (UK average) to 2.02% | 20% | about 25,200 |
| 2.0% to 2.4% | 20% | about 21,000 |
| 5.0% to 5.5% | 10% | about 31,000 |
| 10.0% to 11.0% | 10% | about 14,700 |

**Read a row and stop at the gap.** The relative lift is only a label for the same underlying change: 2.0% to 2.4% is a 20% relative lift and a 0.4 percentage point absolute one. The formula only ever sees the absolute gap.

Three habits make the table usable.

- **Take the baseline from your own page.** That URL, that device mix, that traffic source, that goal. A sitewide average blends your best and worst pages and understates the sample you need.
- **Enter absolute rates.** A calculator that only accepts a percentage uplift hides the number that matters. 10% of 1.68% isn't 10% of 10%.
- **Round up.** 25,200 visitors per variant means 25,200, not "about 25,000 and call it done".

### Why the low-converting shop pays more

Drop from a 10% baseline to a 2% baseline and the visitors needed for the same 10% relative lift rise from about **14,700** to about **80,000** per variant. Drop to the UK average and it's nearer **96,000**. A rate close to 10% leaves statistical room to spare; a rate near 2% leaves almost none, so spotting the same relative change costs several times the traffic.

These numbers assume a two-sided frequentist test, one comparison, an even split and a yes-or-no conversion. Change any of those and the answer moves. A test measuring revenue per visitor rather than orders needs more data than any row above, because order values scatter far more widely than a binomial outcome.

> **The planning check:** if the total exceeds the traffic you can send before your next campaign, don't run the test. Raise the minimum detectable effect, or move the test to a page with more traffic.

## Baseline, MDE, Power and Significance

A calculator is four dials and no fifth. Set all four and the output is arithmetic. Set them carelessly and the output is a number that looks like a plan.

**Baseline conversion rate** is conversions divided by eligible visitors on the control experience. Eligible does real work in that sentence: the same page, the same funnel step, the same source, the same device class. If mobile converts at 1.1% and desktop at 2.6%, a blended 2.0% describes a visitor who never lands.

The four inputs sit around the calculation like dials on a desk.

![A diagram illustrating the four key components of an A/B testing calculator: baseline, MDE, power, and significance.](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/daba081c-13fa-447b-bd2b-e52d6681f286/ab-testing-calculator-ab-testing-metrics.jpg)

**Minimum detectable effect (MDE)** is the smallest true lift you'd still act on, written as an absolute gap such as 0.2 percentage points. Planning often collapses at this point. Teams type the lift they hope for, usually something flattering like 15%, then treat the resulting sample size as a budget rather than a requirement.

### Power and significance are not the same dial

Statistical power is the chance of calling the lift significant if it exists. At 80% power you accept a one-in-five chance of missing a real effect of that size. Push to 90% and the sample grows, because the z value behind power rises from about **0.84** to about **1.28**.

> Power is how often you notice a loaded coin. Significance is how often a fair coin fools you.

Significance is the false-winner risk you accept when there's no real difference, generally 5% on a two-sided test at p under 0.05, because a variant can lose as easily as win. One comparison at 5% isn't the same as three challengers against a single control, where the family of comparisons inflates the odds of a lucky result.

What the output isn't: a promise the observed lift will match the MDE, and certainly not proof the change is worth building. It's the number of observations needed for a fair call.

For the mechanics of building and splitting the test itself, from QA through traffic allocation, [start experimenting with A/B tests](https://www.ecorn.agency/blog/the-complete-tutorial-to-a-b-testing) and come back to size it. Then freeze all four dials before launch. A sample size you can revise mid-test isn't a commitment, it's a moving target, and every extra glance at the dashboard is a slightly higher chance of shipping noise.

## How the Sample Size Formula Works

Two calculations hide inside a calculator: one before the test sizes it, one after the test judges it. Neither needs software to follow.

### Before the test, the sizing formula in words

Take the control rate and the rate you want to detect, and average them. That average is the pooled conversion rate, usually written p-bar. Multiply p-bar by one minus p-bar. Multiply by a constant set by your significance level and power. Divide by the squared absolute gap between the two rates. What's left is visitors per variant for a two-sided test with equal groups.

At 95% two-sided confidence the z value is **1.96**. At 80% power it's **0.84**. Add them, square the sum, double it, and you get **15.68**, the constant. That single number explains why small gaps hurt so much: the gap sits in the denominator, squared.

![A diagram illustrating the mathematical formulas used to calculate sample size for A/B testing statistical significance.](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/aaeacc76-506e-4ce0-b253-5c7e5f2bfcea/ab-testing-calculator-sample-size-formula.jpg)

Run a UK case through it. Control 2.0%, target 2.2%, so the gap is 0.002 and p-bar is 0.021. P-bar times one minus p-bar is about 0.0206. Multiply by 15.68 and you get about 0.322. Divide by 0.002 squared, which is 0.000004, and you land at roughly **80,600 visitors per variant**, matching the 80,000 row once rounded.

### After the test, the z statistic

Once the data is in, the observed gap between the two rates is divided by its standard error. That standard error is the square root of each rate times one minus itself, divided by that variant's sample, added together. If the absolute z exceeds **1.96**, the result clears the 5% two-sided bar.

Two samples, not one. An ecommerce test compares two observed rates, so both variants sit inside the error term. Loosen the bar to 90% confidence, where the cut-off is about **1.64**, and you'll declare winners sooner and be wrong more often. That trade is legal and usually bad.

A calculator is only as honest as the absolute rates you type into it. A blended baseline, a hoped-for MDE and a sample size treated as a target rather than a limit produce a confident-looking number attached to nothing.

## How Long Your Traffic Takes to Reach the Sample

A visitor target with no calendar attached isn't a plan. Divide the visitors per variant by the eligible visitors that variant will receive each day, after the split and after filters such as country, new versus returning, or mobile only.

Say a product page gets 1,000 eligible sessions a day, split evenly, so 500 per variant. Against the 80,000-visitor target from the 2.0% baseline, that's **160 days**. Roughly five months for one comparison.

The confusing part is what the dashboard shows in week three. At a 2.0% rate, 200 conversions per variant arrive in about **20 days** and everything looks healthy. It isn't finished. [UK ecommerce research on test planning](https://edmondscommerce.co.uk/research/ecommerce/) treats roughly **200 to 500 conversions per variant** as a practical requirement at around 80% power against a 95% confidence threshold, and its worked example makes the trap plain: 10,000 visitors at a 2% rate produce about **200 conversions**, the bottom of the useful range, while the same 10,000 visitors at 1% produce about **100** and are too thin to trust. The conversion floor is a minimum. The visitor target governs.

| Visitors per variant per day | Baseline | Time to about 200 conversions |
| --- | --- | --- |
| 500 | 2.0% | about 3 weeks |
| 500 | 1.68% (UK average) | about 24 days |
| 2,500 | 2.0% | under a week |
| 5,000 | 2.0% | 2 days |
| 5,000 | 1.68% (UK average) | about 2.5 days |
| 10,000 | 2.0% | 1 day |

### From daily traffic to a written end date

1. **Pull eligible sessions per day**, not total site traffic, for that exact URL and audience.
2. **Divide by the number of variants** so you know what each one receives.
3. **Multiply by the baseline** to get expected conversions per variant per day.
4. **Divide the visitor target** by daily visitors per variant, then round up to whole weeks.
5. **Cover a full week at minimum**, because weekday and weekend behaviour differ, and checkout goals differ most.
6. **Check conversions** against a few hundred per variant before committing.

If the run needs more than a couple of months, the fix is a bigger change or a busier page. It isn't a shorter test.

The same UK source warns that stopping the moment a dashboard crosses 95% confidence inflates false-positive risk. Predefine the sample or the duration, hold the split exactly as planned for the whole run, stay clear of a sale or a campaign switch, and turn the target into an end date using the [A/B test duration calculator](https://www.otterab.com/free-tools/duration-calculator) before anyone writes a line of code.

## Check Consent Before the Clock Starts

The calculator tells you when you can decide. It says nothing about whether you may collect the data.

[UK guidance on A/B testing and cookie consent](https://dev.cookiepal.io/blog/do-ab-testing-tools-require-cookie-consent) sets out when aggregate experiment data may fall under the statistical-purposes exception rather than requiring consent, and the conditions are narrower than many assume. All four need to hold.

1. **The purpose is statistical.** You're measuring which version of the service performs better, not building an audience.
2. **The results can't identify anyone.** Aggregated counts, not individual-level profiles.
3. **The data isn't repurposed.** No advertising, no profiling, no passing it to a third party for their own use.
4. **Visitors are informed and can object.** A clear notice plus a simple, free route to say no.

Answer each one carefully and record the answers next to the sample size. If any answer is no, the test may need consent or a redesign, and the visitor target is irrelevant until that's settled. Device storage, cross-site tracking, advertising audiences and purchase-linked profiles all pull a test towards the consent side; aggregate counting for a layout decision usually doesn't.

> **Record the decision:** who signed off, on what date, and which of the four conditions you relied on.

Whoever owns privacy signs this off, not the growth team. The longer version of the check sits in the [privacy and consent documentation](https://www.otterab.com/docs/getting-started/privacy-and-consent).

## Two UK Tests Worked End to End

A product-page test and a lead-form test can need very different sample sizes, even when both use the same calculation. For a baseline, the first example uses the **1.68% average UK ecommerce conversion rate for September 2025**, reported in [IRP Commerce's UK ecommerce benchmark data](https://www.irpcommerce.com/en/gb/EcommerceMarketData.aspx).

![A conceptual diagram showing an A/B test comparing a product page against a lead generation form.](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/43da6d24-b3f8-4b21-bea9-e1b7ed1c115d/ab-testing-calculator-ab-test-diagram.jpg)

**Shop one, a product page.** The team will rebuild the layout only if it can detect a **20% relative lift**. That would take the conversion rate from 1.68% to about **2.02%**, a difference of 0.34 percentage points. Using the sample-size assumptions below, the estimate is roughly **25,200 visitors per variant**, or **50,400 in total**.

At 1,000 eligible sessions a day, an even split sends 500 to each variant. Reaching the target takes about **50 days**, so allow eight weeks and check that the test window avoids the peak sale. At 1.68%, 500 sessions produce about **eight orders per variant per day**. A 200-conversion floor would take roughly 25 days, so the visitor target, not the conversion floor, sets the schedule.

### The lead-form test that finishes in a fortnight

For shop two, suppose the baseline is 8% and the team would act on a rise to 10%. With the same 80% power, two-sided 95% confidence and equal allocation, a standard two-proportion calculation gives roughly **3,200 visitors per variant**. At 250 eligible visitors per variant per day, that is about **13 days**.

The 8% baseline and 10% target are example inputs, not universal benchmarks. The result changes with the baseline, the minimum lift worth detecting, and the power and significance settings. A small gap between rates needs more visitors to distinguish from random variation.

The walkthrough below shows the sequence running in a live tool.

<iframe width="100%" style="aspect-ratio: 16 / 9;" src="https://www.youtube.com/embed/6n5I_5JfFHs" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

For either example, enter the control conversion rate, the target rate or absolute gap, power, confidence level, test direction and traffic split. Shop one uses 1.68% and a 0.34-point gap; shop two uses 8% and a 2-point gap. Keep the split fixed during the test. Changing it changes how quickly each variant gathers visitors, and extending the calendar does not make a result reliable if eligible traffic is lower than expected.

These estimates plan the visitor count and calendar, not a guaranteed winner. Before launch, check that the traffic can reach the target, that the consent decision is settled, and that a conversion lift would still make commercial sense when measured against revenue.

## Check Revenue and Lock the Test Before Launch

A cheap call to action can win the conversion test and cost the business money. Push visitors towards a lower-value product, a discount code or a lead magnet and orders climb while average order value and revenue per visitor slide. The test worked. The quarter didn't.

That risk matters at national scale. Online sales represented **27.8% of Great Britain retail sales** in June and July 2025, while online sales values in October 2025 were **4.8% higher** than a year earlier, per the [ONS retail sales bulletin](https://www.ons.gov.uk/businessindustryandtrade/retailindustry/bulletins/retailsales/july2025). At that volume, a metric that rises while revenue stays flat is expensive.

A conversion-rate lift and a revenue-per-visitor lift are two different questions about the same test.

![A performance report showing conversion rate, revenue per visitor, and average order value for A/B testing.](https://cdnimg.co/3716ee4f-bd1a-44a8-ac85-c2df5af21725/04d20ce7-3376-4e99-abd0-79a3ce8a33ec/ab-testing-calculator-performance-metrics.jpg)

Revenue is noisier than a yes-or-no purchase. One large basket swings an average, so a revenue comparison needs more observations than a conversion comparison, and a calculator that only asks for a conversion lift will look finished far too soon. Put conversion rate, average order value and revenue per visitor side by side before you call anything.

Repeated looks inflate false winners too. Every extra glance at a live test is another decision point, and every extra challenger widens the family of comparisons. Otter A/B applies an automatic Bonferroni correction to A/B/n tests and offers free Sample Size, Significance and Duration calculators, so the sizing happens before the build.

### The seven-step lock-in

1. **Pull the baseline** from the same URL, device mix and traffic source you'll test.
2. **Choose an absolute MDE** tied to a revenue or margin change you'd ship.
3. **Keep 80% power and 95% confidence** unless there's a written reason to move them.
4. **Turn visitors per variant into whole weeks** using eligible daily traffic.
5. **Check expected conversions** against a few hundred per variant.
6. **Tighten the bar when you have more than one challenger**, or accept a higher false-winner rate.
7. **Write the stop rule** before launch, and never end on the first day the result looks green.

Run the numbers through the [revenue impact calculator](https://www.otterab.com/free-tools/revenue-impact-calculator) to see what the lift is worth in money rather than orders, and set it against the trading window you're working inside.

> The statistically easiest metric to improve is rarely the one that pays the bills.

One decision follows. If the duration fits the trading window and the revenue maths holds, launch. If not, test a bigger change or move to a busier page. Shrinking the sample doesn't shrink the uncertainty, it hides it.

---

Otter A/B gives you the planning kit to run this properly, with a free Sample Size Calculator, Significance Calculator and Duration Calculator, plus revenue reporting that tracks purchases, average order value and revenue per variant so a conversion win gets checked against the money it brings in. Set your baseline, choose an MDE you'd ship, and let the platform hold you to the sample you planned rather than the first green number you spot. Start free at [Otter A/B](https://www.otterab.com) and size your next test before you build it.

---

Canonical page: https://www.otterab.com/blog/ab-testing-calculator