What can stop a test automatically?

Visitor and conversion caps are available for manual splits and multi-armed bandits. Manual splits can also stop when the primary goal reaches winner status. A bandit does not use winner auto-stop because its new-visitor allocation keeps adapting. The first available condition to trip ends the test.

How does the visitor limit count traffic?

It counts unique human visitors assigned to the test — the same number you see in the visitor column on the results page. Bots, impersonation sessions, and excluded-IP traffic are not counted. The check is total visitors across all variants, not per-variant.

How does the conversion limit count conversions?

It counts unique converters on the primary goal — visitors who converted at least once. Multiple conversions from the same visitor count as one. Conversions on secondary goals don't affect this counter.

What does 'decision threshold reached' actually mean?

On a manual split, auto-stop on winner ends the test when a variant's primary-goal status flips to winner. The score must clear the effective threshold and both that variant and control need at least 30 visitors on the primary goal. Multivariate frequentist tests use the adjusted threshold automatically. Bandits still report Bayesian decision status but do not use it as an auto-stop condition.

When does Otter actually evaluate these conditions?

Every five minutes via a background job. So in the worst case there's about a five-minute window between hitting your stop condition and the test actually transitioning to completed. For high-traffic tests with tight conditions, expect to see the visitor or conversion count slightly over your configured cap — the difference is one batch interval of traffic.

Which condition wins if more than one is set?

The first one that trips. Internally the checker evaluates in this order: visitor limit, then conversion limit, then decision threshold. The test completes once any single condition is satisfied — it doesn't wait for all of them. The stop reason recorded on the test tells you which one triggered.

What if a limit is set to 0 or left blank?

It's ignored. The checker only enforces a limit when the configured value is greater than zero. Leave a field blank to skip that condition entirely; set it to a positive integer to enforce it.

Can I add or change stop conditions while the test is running?

Not while it is running, but you don't have to start over. Pause the test, click Edit, change the stop conditions, and resume — the test keeps every visitor and conversion it has already collected. Editing is available on draft and paused tests; only a running test is locked.

Does auto-stop on winner risk false positives?

For a manual split, yes — it has the same risk as stopping early yourself. A visitor limit is a cap that ends the test, not a floor that holds it open, so the 30-visitor-per-arm minimum is the only automatic guard. Frequentist manual splits should usually run to their planned sample size; Bayesian manual splits are safer to stop at the threshold. Bandits do not expose winner auto-stop.

What happens when auto-stop fires?

The test transitions to completed (the same state as if you'd clicked Complete manually). ended_at is recorded with the moment of transition. The reason is logged ('Visitor limit reached', 'Conversion limit reached', or 'Decision threshold reached') and is visible in the test's activity log. Results are frozen at that point — no new assignments, no new conversions.

Can I resume a test that auto-stopped?

Yes — the same rules as a manually completed test. Resume from the dashboard, and as long as no scheduled end date has passed, the test re-enters running. Stop conditions stay configured, so the next batch of traffic will be re-evaluated and could auto-stop again. Reconsider the conditions before resuming so you don't immediately re-trigger them.

Building Tests5 min read

Stop Conditions

Set visitor or conversion caps on any experiment, or winner auto-stop on a manual split.

Browse docs

Manual splits can stop on a visitor cap, conversion cap, or winner. Multi-armed bandits can use the two caps but do not use winner auto-stop.

Stop conditions are the hands-off way to manage a running test. Set the rules upfront, launch, and Otter auto-completes the test when any rule trips. The test transitions to the same completed state you'd get from clicking Complete manually — including the same ended-at timestamp and the same frozen-results behavior.

You don't have to use stop conditions. A test without any configured stop conditions runs until you complete it manually or hit a scheduled end date. Most teams set at least one — usually a visitor cap — to keep tests from running longer than they need to. Every condition here ends a test; none of them holds a test open until it reaches a sample size.

Multi-armed bandits do not use winner auto-stop. Their new-visitor allocation keeps adapting as results arrive. Visitor and conversion caps remain available when you need a firm operational limit.

Available conditions

Visitor limit

visitor_limit

Stop when total unique human visitors assigned to the test reaches this number.

Counts unique humans (bots, impersonation sessions, and excluded-IP traffic are excluded). The count is total across all variants, not per-variant.

Conversion limit

conversion_limit

Stop when total unique converters on the primary goal reaches this number.

Counts visitors who converted at least once on the primary goal. Multiple conversions from the same visitor count as one. Secondary goals don't affect this counter.

Auto-stop on winner (manual splits only)

auto_stop_on_winner

Stop when the primary goal has a variant in 'winner' status, against the test's effective confidence threshold.

Uses the same decision logic as the results page — Bayesian or frequentist with the configured confidence level, Bonferroni-adjusted for multivariate frequentist tests. Winner status additionally requires at least 30 visitors on both the control and the winning variant, so a test cannot auto-stop on a thin sample however high the score reads.

Evaluation rules

Checked every 5 minutes via a background job. Expect a small lag — up to one batch interval of traffic — between hitting a limit and the test transitioning to completed.
First-to-trip wins. Internally the checker evaluates in this order: visitor limit, then conversion limit, then decision threshold. As soon as one is satisfied, the test stops; it doesn't wait for the others.
Zero or blank disables a condition. Only positive integers count. Leave a field blank to skip the condition entirely.
The reason is recorded. The test's activity log captures whether it stopped on visitor limit, conversion limit, or decision threshold so future-you knows why it ended when it did.
Winner status has a 30-visitor floor per arm. On a manual split, the decision-threshold condition can only trip once both the control and the winning variant have at least 30 visitors on the primary goal. Until then the variant stays at Collecting data and the test keeps running, even when the score is already past your threshold — see Reading Results.
Editing conditions means pausing first. Stop conditions can't be edited while a test is running. Pause the test, click Edit, change the conditions, and resume — collected data is kept. Only draft and paused tests are editable.

How to pick good stop conditions

A visitor limit is a cap, not a floor. It ends the test when the count is reached, so it cannot stop a manual split's winner auto-stop from calling a thin sample — adding one only makes the test stop sooner. The 30-visitor-per-arm floor is the only automatic guard. Past that, stopping a frequentist manual split on power means watching the wizard's sample-size estimate and completing the test yourself.

Don't make limits too tight on low-traffic tests. A 1000-visitor cap on a page that gets 30 visitors per day means a long-running test. For a frequentist manual split, use the wizard's sample-size advisor to ground the cap. For Bayesian tests, choose an operational cap you can justify.

Re-think conditions before resuming. If a test auto-stopped and you resume it, the same conditions will be re-evaluated against the same traffic that tripped them. Usually you want to raise the cap or remove the condition before resuming, otherwise you'll just re-trigger it.

Bayesian manual splits handle peeking better. If you're uncomfortable with the auto-stop-on-winner risk for frequentist, run the manual split in Bayesian mode. Bandits already use Bayesian reporting and do not expose winner auto-stop.

Frequently asked questions

Quick answers to the questions teams ask most about this part of Otter.