Which analysis method should I choose?

For a manual split, frequentist is the classic default; choose Bayesian when you want probability-based reporting or safer peeking. Multi-armed bandits use Bayesian reporting automatically because allocation adapts from the same primary-goal results.

What confidence level should I configure?

95% is the industry standard and what most teams should pick. Use 90% if you're running rapid iteration tests on lower-stakes changes (button copy, minor layout) and want to ship more frequently. Use 99% for high-stakes changes (pricing pages, checkout flow) where the cost of a false positive is high. 80% is rarely the right answer.

Why does my multivariate test need a higher threshold?

When you run multiple challenger variants, the chance that at least one of them looks like a winner by random chance increases with each variant. The Bonferroni correction divides your alpha (1 - confidence_level) by the number of challengers, raising the per-variant bar. A 95% test with two challengers needs each variant to clear 97.5%, not 95%. Bayesian tests don't apply this correction — their probabilistic interpretation handles multiplicity naturally.

What does 'Chance to Beat Original' mean in Bayesian mode?

It's the probability that the variant's true conversion rate (or revenue per visitor) is higher than the control's, given the data we've observed so far. If it's 97%, there's a 97% chance variant beats control and a 3% chance control beats variant. When the variant is currently behind, the label flips to 'Chance Original Beats Variant' so the percentage stays positive and interpretable.

What is the 'effective confidence threshold' I see on the results page?

It's the actual score your test needs to clear to be declared a winner, after any adjustments. For single-variant frequentist tests, it equals your configured confidence level. For multivariate frequentist tests, it's the Bonferroni-adjusted threshold. For Bayesian tests, it equals your configured confidence level (no adjustment). The progress bar on the results page fills relative to this threshold.

How do I know when a test is done?

For a frequentist manual split, read the decision score together with the wizard's planned sample size and confidence intervals. A Bayesian manual split has no fixed-horizon estimate and is safer to monitor continuously. A bandit also has no fixed-horizon estimate: watch its Bayesian results and allocation panel, then stop manually or use a visitor or conversion cap.

My score is above the threshold but the variant still says 'collecting data' — why?

A variant cannot be declared a winner until both the control and that variant have at least 30 visitors per arm on the goal being scored. Below that floor Otter still calculates and shows the score. On a manual split, winner auto-stop therefore cannot fire until both arms reach 30; bandits do not use winner auto-stop.

Why is my conversion rate different from my analytics tool?

Otter counts conversions attributed to a single variant per visitor, deduplicated within the test window. Analytics tools usually count every event, including repeat conversions from the same person, and don't filter for variant assignment. The two numbers won't match exactly — they're measuring different things.

Can I peek at the results before the test is done?

Yes. A Bayesian manual split is safer to monitor continuously than a frequentist one, where stopping on an early peek inflates false positives. Bandits already use Bayesian reporting and update their new-visitor allocation as results arrive.

Can I see how my results changed over time?

Yes. The chart on the results page has over-time views — switch it to "Lift over time" to watch each version's lead grow or shrink, or "Decision score over time" to see your confidence build. You can view it day-by-day or hour-by-hour, and toggle between a running total (smoother) and each period on its own (more detailed). It's a great way to check a result is holding steady, not just having a lucky day.

What does a 'Noisy trend' note mean?

Above the over-time chart, Otter gives the trend a quick read: "Steady trend" means the line is calm and the result looks consistent; "Some movement" means it's wobbling a bit; and "Noisy trend" means the numbers are bouncing around a lot. A noisy trend is a nudge to collect more data before trusting the result — early on, big swings are normal and usually settle as more visitors arrive.

Analyzing Results8 min read

Reading Results

Frequentist and Bayesian analysis, confidence thresholds, and key fields.

Browse docs

Manual-split experiments can use frequentist or Bayesian reporting. Multi-armed bandits use Bayesian reporting automatically. Every results page exposes the resolved decision score for its method.

A results page in Otter answers three questions: is there a difference between the variants, how big is it, and how sure are we? Each of those questions has its own column on the table — the score, the lift (improvement), and the confidence interval — and the three are connected. A big lift with tight intervals and a high score is a clear win. A small lift with wide intervals and a borderline score is noise.

For a manual split, the most important thing to understand is which method you chose. Frequentist asks “if there were no real difference, how surprised would I be to see data this extreme?” Bayesian asks “what's the probability the variant is genuinely better?” A bandit fixes that choice to Bayesian because its traffic strategy learns from the same primary-goal results.

Frequentist

Significance (1-p)

The classic A/B testing approach, and the right default for most teams. Otter automatically picks the right calculation for your data — a careful one when numbers are small, the standard one for typical conversion rates, and a revenue-aware one for money — so you never choose a test yourself. The score shows how confident the result is, as a percentage. (For the statistically-minded: Fisher's exact, a two-proportion z-test, and Welch's t-test respectively, with the score equal to one minus the p-value.)

•Score label on the results page: "Significance (1-p)".
•You hit "significance" when the score reaches your effective confidence threshold (see below).
•Multivariate tests automatically apply a Bonferroni correction — the bar to clear rises with the number of challenger variants.

Bayesian

Chance to Beat Original

A probability-based approach that answers "what is the chance this variant is better than the control?" Otter runs a Bayesian bootstrap and reports the probability that each variant outperforms the original. Revenue metrics use the same bootstrap on the underlying sample distributions.

•Score label on the results page: "Chance to Beat Original".
•If a variant is currently behind, the label flips to "Chance Original Beats Variant" and the percentage shown is the chance the control wins.
•The configured confidence level is used directly as the threshold — there is no Bonferroni adjustment for Bayesian analysis.

Multi-armed bandit results

Bandits start with an even split, then refresh the recommended new-visitor allocation about every five minutes from primary-goal results. Existing visitors keep their first version. The results page shows the Bayesian decision score plus the recommended and actual allocations. Bandits do not use a fixed-horizon sample-size estimate or winner auto-stop; visitor and conversion caps remain available.

Effective confidence threshold (Frequentist only)

When you run a frequentist test with more than one challenger variant, Otter raises the confidence threshold using a Bonferroni correction so the chance of any single variant looking like a winner by accident stays bounded. The formula is:

effective_threshold = (1 - α / challenger_count) × 100
  where α = 1 - (confidence_level / 100)

For example, a test with 95% confidence and 2 challenger variants (3 variants total) needs the score to clear 97.5%, not 95%, before declaring a winner. Bayesian tests use the configured confidence level as-is — no adjustment is applied. The results page always shows the effective threshold you actually need to beat, not just what you configured.

Minimum data before a variant can win

Otter holds every variant at Collecting data until both the control and that variant have at least 30 visitors on the goal being scored. Below that floor no variant is labelled a winner, however high the score climbs.

The floor is per arm, not per test. Control 40 / Variant 12 does not clear it — the variant arm is short.
It applies to both analysis methods, and to each goal independently. A goal with enough traffic is scored while a thinner goal on the same test is not.
Revenue goals count the same way: every assigned visitor contributes a sample, zero included, so the floor is still 30 visitors per arm.
The score is still calculated and shown below the floor. A Bayesian test can read “82% chance to beat” while the status is still Collecting data — that is expected, not a bug.

On manual splits, this matters most with auto-stop-on-winner, which fires on winner status: a test cannot auto-stop until both arms clear 30, no matter what the score says.

Common fields on the results page

Visitors — unique humans assigned to each variant (bot traffic and impersonation sessions are excluded).
Conversions — primary-goal conversions attributed to each variant. Secondary goals are reported separately and do not affect the score.
Conversion rate / Revenue per visitor — the metric family the primary goal drives. Revenue goals switch to revenue-per-visitor; everything else uses conversion rate.
Improvement — relative lift over the control. Can go negative when the variant underperforms.
Score — the resolved decision score described above. The progress bar fills relative to the effective threshold, capping at 100%.

Reading results well

Don't call a frequentist manual split early. Read its score against the sample size the wizard estimated and let the test reach it, so you stop on power rather than on a peek. A visitor limit is a cap that ends a test early, not a floor that holds it open.

Pay attention to the lift, not just the score. A test can hit a high score with a 0.3% lift if the sample is huge — statistically real, practically irrelevant. Ask whether the lift is large enough to matter to the business before you ship.

If results look surprising, check the activity log first. Pauses, edits, and stop-condition trips can all explain unexpected patterns. The activity log on the test page captures what happened and when.

Use Bayesian for a manual split when you want to peek safely. Its “chance to beat control” reading is safer to monitor mid-test than a frequentist p-value. Bandits already use Bayesian reporting while their traffic allocation adapts.

Frequently asked questions

Quick answers to the questions teams ask most about this part of Otter.