How does Otter decide which variant a visitor sees?

When a new visitor lands, the SDK hashes the visitor UUID and test ID into the current allocation. Manual splits keep configured weights; multi-armed bandits periodically adapt new-visitor weights from primary-goal results. The chosen assignment is persisted, so that visitor keeps the same version when they return.

What if a visitor switches devices or browsers?

By default, they get reassigned because the visitor UUID lives in a cookie. If you call optimo.userID('some_id') with a stable user identifier (after login, say), the assignment becomes user-scoped: the same user gets the same variant across devices, browsers, and incognito sessions. Without userID, anonymous device-level assignment is the model.

Are assignments shared across tests?

No. Each test gets an independent assignment driven by a different hash seed. A visitor who lands on variant B for test #1 has independent odds of landing on any variant for test #2. This avoids any "groove" where a visitor is consistently assigned to (e.g.) the second variant across multiple tests.

Which statistical test does Otter actually run?

It depends on the data: Fisher's exact test when conversion counts are sparse ( control).

Why use Fisher's exact test instead of just a z-test?

The z-test relies on a normal approximation that breaks down when conversion counts are very small. With only a handful of conversions per variant, the approximation can report 'significant' results that aren't real. Fisher's exact test enumerates the actual probability and gives you a correct p-value even at small sample sizes. Otter switches automatically when it detects sparse counts.

What is the anti-flicker snippet doing?

It's a single inline tag that sets body { opacity: 0 } so the page is invisible while the SDK fetches and applies variants. As soon as the SDK has resolved which variants to apply (typically a few hundred milliseconds), it removes the style and reveals the page. This avoids the 'flash of unstyled control' problem where visitors see the original version flash before the variant snaps in. A 3-second failsafe guarantees the page becomes visible even if the SDK fails to load.

How is bot traffic excluded?

Server-side, every SDK request is checked against known bot signatures (user-agent patterns and IP lists). Bot sessions are still recorded in raw logs so you can debug, but they're excluded from variant assignment, conversion counts, and the results page. Impersonation sessions (where a logged-in staff member 'becomes' a user) are similarly excluded so internal QA traffic doesn't pollute results.

Can a visitor be assigned to a test but never convert because they leave?

Yes — and that's the correct behavior. The visitor's denominator (visits) increments at assignment, not at conversion. A variant that gets lots of visitors but few conversions has a low conversion rate, which is exactly what you want to know about. This is also why bot exclusion matters — bot visits would otherwise dilute the denominator.

Analyzing Results6 min read

Concepts

Variant assignment, statistical significance, and the anti-flicker snippet.

Browse docs

The handful of ideas you need in your head to use Otter confidently — how variants are assigned, how the math actually works, and how the SDK avoids flicker.

You don't need a maths degree to use Otter — but three simple ideas explain why your results look the way they do and how much to trust them. Here they are in plain language. (If you love the detail, the box at the bottom names the exact methods.)

Variant Assignment

Each visitor is sorted into one version and stays there. Manual splits use the configured weights for new assignments. Multi-armed bandits adapt those new-visitor weights from primary-goal results. Either way, the same visitor sees the same version when they return.

Statistical Significance

This is how we tell a real difference from random luck. Manual splits let you choose frequentist or Bayesian analysis. Multi-armed bandits use Bayesian analysis automatically because their allocation learns from the same primary-goal results. Otter still picks the metric-specific calculation for conversion and revenue data.

Anti-Flicker

Stops visitors from glimpsing the original page for a split second before your version loads — that flash is called "flicker." The snippet briefly keeps the page hidden until the right version is ready, then reveals it (usually in a few hundred milliseconds). A 3-second safety timer makes sure the page always appears, even if something goes wrong.

Why this matters in practice

Deterministic assignment means you can A/B everything safely. A visitor never bounces between variants between page loads. If you see someone “flip” in session replay, it's almost always because they cleared cookies or switched browsers.

Test selection adapts to your data. Otter picks Fisher's exact test for sparse counts and a z-test for typical comparisons automatically — you don't need to think about which to use. Revenue gets Welch's t-test because revenue distributions are heavy-tailed and a z-test would lie to you.

Anti-flicker is non-negotiable. If visitors see the original page flash before the variant, they've already responded to the original — your test is measuring noise. The 3-second failsafe keeps the page from disappearing entirely if the SDK ever fails to load, but the normal case is a few hundred milliseconds.

Frequently asked questions

Quick answers to the questions teams ask most about this part of Otter.