Question 1

How does Otter A/B decide which variant a visitor sees?

Accepted Answer

When a new visitor lands on a test, the SDK generates a UUID and stores it in a first-party cookie. The visitor UUID and the test ID are hashed together to produce a number in [0, 1). That number is mapped to a variant via the configured traffic split. Because the inputs are deterministic, the same visitor on the same test always gets the same variant — even after cache clears, because the assignment is also persisted server-side.

Question 2

What if a visitor switches devices or browsers?

Accepted Answer

By default, they get reassigned because the visitor UUID lives in a cookie. If you call optimo.userID('some_id') with a stable user identifier (after login, say), the assignment becomes user-scoped: the same user gets the same variant across devices, browsers, and incognito sessions. Without userID, anonymous device-level assignment is the model.

Question 3

Are assignments shared across tests?

Accepted Answer

No. Each test gets an independent assignment driven by a different hash seed. A visitor who lands on variant B for test #1 has independent odds of landing on any variant for test #2. This avoids any "groove" where a visitor is consistently assigned to (e.g.) the second variant across multiple tests.

Question 4

Which statistical test does Otter A/B actually run?

Accepted Answer

It depends on the data: Fisher's exact test when conversion counts are sparse (< 5 in any cell), giving you valid p-values even at small sample sizes; a two-proportion z-test for typical conversion-rate comparisons when counts are larger; and Welch's t-test with Satterthwaite degrees of freedom for revenue metrics. Bayesian mode replaces all of these with a Bayesian bootstrap that estimates the posterior distribution of each variant's metric and computes P(variant > control).

Question 5

Why use Fisher's exact test instead of just a z-test?

Accepted Answer

The z-test relies on a normal approximation that breaks down when conversion counts are very small. With only a handful of conversions per variant, the approximation can report 'significant' results that aren't real. Fisher's exact test enumerates the actual probability and gives you a correct p-value even at small sample sizes. Otter A/B switches automatically when it detects sparse counts.

Question 6

What is the anti-flicker snippet doing?

Accepted Answer

It's a single inline <style> tag that sets body { opacity: 0 } so the page is invisible while the SDK fetches and applies variants. As soon as the SDK has resolved which variants to apply (typically a few hundred milliseconds), it removes the style and reveals the page. This avoids the 'flash of unstyled control' problem where visitors see the original version flash before the variant snaps in. A 3-second failsafe guarantees the page becomes visible even if the SDK fails to load.

Question 7

How is bot traffic excluded?

Accepted Answer

Server-side, every SDK request is checked against known bot signatures (user-agent patterns and IP lists). Bot sessions are still recorded in raw logs so you can debug, but they're excluded from variant assignment, conversion counts, and the results page. Impersonation sessions (where a logged-in staff member 'becomes' a user) are similarly excluded so internal QA traffic doesn't pollute results.

Question 8

Can a visitor be assigned to a test but never convert because they leave?

Accepted Answer

Yes — and that's the correct behavior. The visitor's denominator (visits) increments at assignment, not at conversion. A variant that gets lots of visitors but few conversions has a low conversion rate, which is exactly what you want to know about. This is also why bot exclusion matters — bot visits would otherwise dilute the denominator.

Concepts

Concepts

Variant Assignment

Statistical Significance

Anti-Flicker

Why this matters in practice

Frequently asked questions