Question 1

What can stop a test automatically?

Accepted Answer

Three conditions, set per test: a visitor cap (stop when N humans have been assigned), a conversion cap (stop when N unique visitors have converted on the primary goal), and a decision-threshold trigger (stop when the primary goal has a variant in 'winner' status). You can configure any combination; the first one to trip ends the test.

Question 2

How does the visitor limit count traffic?

Accepted Answer

It counts unique human visitors assigned to the test — the same number you see in the visitor column on the results page. Bots, impersonation sessions, and excluded-IP traffic are not counted. The check is total visitors across all variants, not per-variant.

Question 3

How does the conversion limit count conversions?

Accepted Answer

It counts unique converters on the primary goal — visitors who converted at least once. Multiple conversions from the same visitor count as one. Conversions on secondary goals don't affect this counter.

Question 4

What does 'decision threshold reached' actually mean?

Accepted Answer

Otter A/B's results engine assigns each variant a status — winner, loser, inconclusive, etc. — based on the primary goal's score against the effective confidence threshold. Auto-stop on winner ends the test when any variant's status flips to winner. Multivariate tests get the Bonferroni-adjusted threshold applied automatically, so the bar is appropriately stricter when you're testing several challengers.

Question 5

When does Otter A/B actually evaluate these conditions?

Accepted Answer

Every five minutes via a background job. So in the worst case there's about a five-minute window between hitting your stop condition and the test actually transitioning to completed. For high-traffic tests with tight conditions, expect to see the visitor or conversion count slightly over your configured cap — the difference is one batch interval of traffic.

Question 6

Which condition wins if more than one is set?

Accepted Answer

The first one that trips. Internally the checker evaluates in this order: visitor limit, then conversion limit, then decision threshold. The test completes once any single condition is satisfied — it doesn't wait for all of them. The stop reason recorded on the test tells you which one triggered.

Question 7

What if a limit is set to 0 or left blank?

Accepted Answer

It's ignored. The checker only enforces a limit when the configured value is greater than zero. Leave a field blank to skip that condition entirely; set it to a positive integer to enforce it.

Question 8

Can I add or change stop conditions while the test is running?

Accepted Answer

You can't edit the test directly once it leaves draft — the wizard is locked. Stop conditions are part of that lock. If you need to change them mid-flight, your options are: complete the test (manually), duplicate it as a new draft with updated conditions, and relaunch; or contact support if there's a specific reason to override the lock for safety.

Question 9

Does auto-stop on winner risk false positives?

Accepted Answer

Yes, slightly — and that's the same risk as stopping early manually. The decision threshold is the same one shown on the results page; if you configured 95% confidence, the test stops at 95%. Stopping the moment confidence is reached is a recognized cause of false positives in frequentist mode. Set a sample-size floor (via the visitor limit) alongside auto-stop-on-winner to mitigate this, or use Bayesian analysis where stopping at the threshold is mathematically safer.

Question 10

What happens when auto-stop fires?

Accepted Answer

The test transitions to completed (the same state as if you'd clicked Complete manually). ended_at is recorded with the moment of transition. The reason is logged ('Visitor limit reached', 'Conversion limit reached', or 'Decision threshold reached') and is visible in the test's activity log. Results are frozen at that point — no new assignments, no new conversions.

Question 11

Can I resume a test that auto-stopped?

Accepted Answer

Yes — the same rules as a manually completed test. Resume from the dashboard, and as long as no scheduled end date has passed, the test re-enters running. Stop conditions stay configured, so the next batch of traffic will be re-evaluated and could auto-stop again. Reconsider the conditions before resuming so you don't immediately re-trigger them.

Stop Conditions

Stop Conditions

The three conditions

Visitor limit

Conversion limit

Auto-stop on winner

Evaluation rules

How to pick good stop conditions

Frequently asked questions