# Reading Results Frequentist and Bayesian analysis, confidence thresholds, and key fields. Every test in Otter is scored using one of two analysis methods, configured per-test alongside a confidence level (80, 90, 95, or 99). The results page shows a single decision score whose label and meaning depend on the method you chose. A results page in Otter answers three questions: is there a difference between the variants, how big is it, and how sure are we? Each of those questions has its own column on the table — the score, the lift (improvement), and the confidence interval — and the three are connected. A big lift with tight intervals and a high score is a clear win. A small lift with wide intervals and a borderline score is noise. The most important thing to understand is what method is scoring your test. Frequentist and Bayesian both work; they answer slightly different questions. Frequentist asks "if there were no real difference, how surprised would I be to see data this extreme?" Bayesian asks "what's the probability the variant is genuinely better?" Most teams use frequentist; teams that run many low-traffic tests or value an intuitive interpretation often prefer Bayesian. ## Analysis methods ### Frequentist Score label: `Significance (1-p)` The classic A/B testing approach, and the right default for most teams. Otter automatically picks the right calculation for your data — a careful one when numbers are small, the standard one for typical conversion rates, and a revenue-aware one for money — so you never choose a test yourself. The score shows how confident the result is, as a percentage. (For the statistically-minded: Fisher's exact, a two-proportion z-test, and Welch's t-test respectively, with the score equal to one minus the p-value.) - Score label on the results page: "Significance (1-p)". - You hit "significance" when the score reaches your effective confidence threshold (see below). - Multivariate tests automatically apply a Bonferroni correction — the bar to clear rises with the number of challenger variants. ### Bayesian Score label: `Chance to Beat Original` A probability-based approach that answers "what is the chance this variant is better than the control?" Otter runs a Bayesian bootstrap and reports the probability that each variant outperforms the original. Revenue metrics use the same bootstrap on the underlying sample distributions. - Score label on the results page: "Chance to Beat Original". - If a variant is currently behind, the label flips to "Chance Original Beats Variant" and the percentage shown is the chance the control wins. - The configured confidence level is used directly as the threshold — there is no Bonferroni adjustment for Bayesian analysis. ## Effective confidence threshold (Frequentist only) When you run a frequentist test with more than one challenger variant, Otter raises the confidence threshold using a Bonferroni correction so the chance of any single variant looking like a winner by accident stays bounded. The formula is: ```text effective_threshold = (1 - α / challenger_count) × 100 where α = 1 - (confidence_level / 100) ``` For example, a test with 95% confidence and 2 challenger variants (3 variants total) needs the score to clear 97.5%, not 95%, before declaring a winner. Bayesian tests use the configured confidence level as-is — no adjustment is applied. The results page always shows the effective threshold you actually need to beat, not just what you configured. ## Common fields on the results page - **Visitors** — unique humans assigned to each variant (bot traffic and impersonation sessions are excluded). - **Conversions** — primary-goal conversions attributed to each variant. Secondary goals are reported separately and do not affect the score. - **Conversion rate / Revenue per visitor** — the metric family the primary goal drives. Revenue goals switch to revenue-per-visitor; everything else uses conversion rate. - **Improvement** — relative lift over the control. Can go negative when the variant underperforms. - **Score** — the resolved decision score described above. The progress bar fills relative to the effective threshold, capping at 100%. > **Reading results well** > > **Don't call it early.** Watching the score creep up on day two and shipping the "winner" is the most common way to ship a false positive. Combine the score with a visitor floor (a sample-size cap from the wizard) so you stop on power, not on a peek. > > **Pay attention to the lift, not just the score.** A test can hit a high score with a 0.3% lift if the sample is huge — statistically real, practically irrelevant. Ask whether the lift is large enough to matter to the business before you ship. > > **If results look surprising, check the activity log first.** Pauses, edits, and stop-condition trips can all explain unexpected patterns. The activity log on the test page captures what happened and when. > > **Use Bayesian when you want to peek safely.** Bayesian's "chance to beat control" reading is much safer to monitor mid-test than a frequentist p-value, which inflates false positives every time you check. ## Frequently asked questions ### Which analysis method should I choose? Frequentist is the right default for most teams — it's the classic A/B testing approach, well-understood, and what most stats training assumes. Choose Bayesian if you want a more intuitive interpretation ("there is a 95% chance variant B beats control"), if you run lots of low-traffic tests where peeking matters, or if you're already running Bayesian elsewhere and want consistency. ### What confidence level should I configure? 95% is the industry standard and what most teams should pick. Use 90% if you're running rapid iteration tests on lower-stakes changes (button copy, minor layout) and want to ship more frequently. Use 99% for high-stakes changes (pricing pages, checkout flow) where the cost of a false positive is high. 80% is rarely the right answer. ### Why does my multivariate test need a higher threshold? When you run multiple challenger variants, the chance that at least one of them looks like a winner by random chance increases with each variant. The Bonferroni correction divides your alpha (1 - confidence_level) by the number of challengers, raising the per-variant bar. A 95% test with two challengers needs each variant to clear 97.5%, not 95%. Bayesian tests don't apply this correction — their probabilistic interpretation handles multiplicity naturally. ### What does 'Chance to Beat Original' mean in Bayesian mode? It's the probability that the variant's true conversion rate (or revenue per visitor) is higher than the control's, given the data we've observed so far. If it's 97%, there's a 97% chance variant beats control and a 3% chance control beats variant. When the variant is currently behind, the label flips to 'Chance Original Beats Variant' so the percentage stays positive and interpretable. ### What is the 'effective confidence threshold' I see on the results page? It's the actual score your test needs to clear to be declared a winner, after any adjustments. For single-variant frequentist tests, it equals your configured confidence level. For multivariate frequentist tests, it's the Bonferroni-adjusted threshold. For Bayesian tests, it equals your configured confidence level (no adjustment). The progress bar on the results page fills relative to this threshold. ### How do I know when a test is done? Three signals: (1) the score reaches the effective confidence threshold; (2) you've hit the sample-size estimate the wizard provided; (3) the conversion-rate confidence intervals are tight enough that you can act on the result. Stopping the moment the score crosses the threshold is fine in Bayesian mode but risks false positives in frequentist mode unless you've also hit the sample size. ### Why is my conversion rate different from my analytics tool? Otter counts conversions attributed to a single variant per visitor, deduplicated within the test window. Analytics tools usually count every event, including repeat conversions from the same person, and don't filter for variant assignment. The two numbers won't match exactly — they're measuring different things. ### Can I peek at the results before the test is done? Yes, but with caveats. In Bayesian mode, peeking is mathematically safe — the probability interpretation doesn't degrade. In frequentist mode, peeking and stopping early inflates false positives. If you peek in frequentist mode, commit to either letting it run to the sample size or switching to Bayesian for that test. ### Can I see how my results changed over time? Yes. The chart on the results page has over-time views — switch it to "Lift over time" to watch each version's lead grow or shrink, or "Decision score over time" to see your confidence build. You can view it day-by-day or hour-by-hour, and toggle between a running total (smoother) and each period on its own (more detailed). It's a great way to check a result is holding steady, not just having a lucky day. ### What does a 'Noisy trend' note mean? Above the over-time chart, Otter gives the trend a quick read: "Steady trend" means the line is calm and the result looks consistent; "Some movement" means it's wobbling a bit; and "Noisy trend" means the numbers are bouncing around a lot. A noisy trend is a nudge to collect more data before trusting the result — early on, big swings are normal and usually settle as more visitors arrive. --- Canonical page: https://www.otterab.com/docs/analyzing-results/reading-results