How does Otter A/B decide which variant a visitor sees?

When a new visitor lands on a test, the SDK generates a UUID and stores it in a first-party cookie. The visitor UUID and the test ID are hashed together to produce a number in [0, 1). That number is mapped to a variant via the configured traffic split. Because the inputs are deterministic, the same visitor on the same test always gets the same variant — even after cache clears, because the assignment is also persisted server-side.

What if a visitor switches devices or browsers?

By default, they get reassigned because the visitor UUID lives in a cookie. If you call optimo.userID('some_id') with a stable user identifier (after login, say), the assignment becomes user-scoped: the same user gets the same variant across devices, browsers, and incognito sessions. Without userID, anonymous device-level assignment is the model.

Are assignments shared across tests?

No. Each test gets an independent assignment driven by a different hash seed. A visitor who lands on variant B for test #1 has independent odds of landing on any variant for test #2. This avoids any "groove" where a visitor is consistently assigned to (e.g.) the second variant across multiple tests.

Which statistical test does Otter A/B actually run?

It depends on the data: Fisher's exact test when conversion counts are sparse ( control).

Why use Fisher's exact test instead of just a z-test?

The z-test relies on a normal approximation that breaks down when conversion counts are very small. With only a handful of conversions per variant, the approximation can report 'significant' results that aren't real. Fisher's exact test enumerates the actual probability and gives you a correct p-value even at small sample sizes. Otter A/B switches automatically when it detects sparse counts.

What is the anti-flicker snippet doing?

It's a single inline tag that sets body { opacity: 0 } so the page is invisible while the SDK fetches and applies variants. As soon as the SDK has resolved which variants to apply (typically a few hundred milliseconds), it removes the style and reveals the page. This avoids the 'flash of unstyled control' problem where visitors see the original version flash before the variant snaps in. A 3-second failsafe guarantees the page becomes visible even if the SDK fails to load.

How is bot traffic excluded?

Server-side, every SDK request is checked against known bot signatures (user-agent patterns and IP lists). Bot sessions are still recorded in raw logs so you can debug, but they're excluded from variant assignment, conversion counts, and the results page. Impersonation sessions (where a logged-in staff member 'becomes' a user) are similarly excluded so internal QA traffic doesn't pollute results.

Can a visitor be assigned to a test but never convert because they leave?

Yes — and that's the correct behavior. The visitor's denominator (visits) increments at assignment, not at conversion. A variant that gets lots of visitors but few conversions has a low conversion rate, which is exactly what you want to know about. This is also why bot exclusion matters — bot visits would otherwise dilute the denominator.

Analyzing Results6 min read

Concepts

Variant assignment, statistical significance, and the anti-flicker snippet.

Browse docs

Concepts

The handful of ideas you need in your head to use Otter A/B confidently — how variants are assigned, how the math actually works, and how the SDK avoids flicker.

You don't need a maths degree to use Otter A/B — but three simple ideas explain why your results look the way they do and how much to trust them. Here they are in plain language. (If you love the detail, the box at the bottom names the exact methods.)

Variant Assignment

Each visitor is sorted into one version and stays there. The choice is random across all your visitors, but it never changes for any one person — we remember it, so the same visitor sees the same version every time they come back, even days later or on a new page.

Statistical Significance

This is how we tell a real difference from random luck. You choose how the math is done: the standard way asks how unlikely your result would be if the versions were really equal, while the "Bayesian" way gives the chance a version truly beats the original. Either way, Otter A/B picks the right calculation for your data automatically — including a special one for money, which behaves differently from simple yes/no conversions.

Anti-Flicker

Stops visitors from glimpsing the original page for a split second before your version loads — that flash is called "flicker." The snippet briefly keeps the page hidden until the right version is ready, then reveals it (usually in a few hundred milliseconds). A 3-second safety timer makes sure the page always appears, even if something goes wrong.

Why this matters in practice

Deterministic assignment means you can A/B everything safely. A visitor never bounces between variants between page loads. If you see someone “flip” in session replay, it's almost always because they cleared cookies or switched browsers.

Test selection adapts to your data. Otter A/B picks Fisher's exact test for sparse counts and a z-test for typical comparisons automatically — you don't need to think about which to use. Revenue gets Welch's t-test because revenue distributions are heavy-tailed and a z-test would lie to you.

Anti-flicker is non-negotiable. If visitors see the original page flash before the variant, they've already responded to the original — your test is measuring noise. The 3-second failsafe keeps the page from disappearing entirely if the SDK ever fails to load, but the normal case is a few hundred milliseconds.

Frequently asked questions

Quick answers to the questions teams ask most about this part of Otter A/B.