What is the Experiment health card?

It's a checklist on the results page that runs the safeguards appropriate to that experiment method. Frequentist manual splits get sample-size and fixed-weight traffic checks. Bayesian manual splits omit the fixed sample target, and bandits omit both fixed-horizon sample-size and fixed-weight traffic-split checks.

What do the colours and words mean?

Each check gets one of four statuses. Green ("all clear") means that check looks healthy. Yellow ("warning") means something is worth a look before you trust the result. Red ("issue") means stop — something is likely wrong and the numbers can't be trusted until you fix it. Blue ("notice") is just information, not a problem. The card's headline always reflects the most serious thing it found.

What does the 'sample size' check tell me?

On a frequentist manual split, it compares collected visitors with the planned target and estimates how many days remain. Bayesian tests have no fixed-horizon target, so they do not show this check.

It says 'confident winner before full sample' — can I ship now?

That notice appears on a frequentist manual split. You can ship, but stopping at the first good reading makes a fluke more likely than the score suggests. The safest move is to let the test reach its planned visitors.

What is a 'sample ratio mismatch' and why is it red?

On a manual split, this compares actual visitors with the fixed weights you configured. A large mismatch usually points to broken delivery, redirects, or filtering, so fix the cause and restart. Bandits intentionally change new-visitor weights and therefore do not run this fixed-weight check.

Why does the traffic-split check sometimes say 'not enough data'?

On a manual split, the fixed-weight check waits until about 200 visitors because an early wobble is normal randomness. Bandits omit this check entirely because their allocation is adaptive.

What does the 'secondary goals' check watch for?

A version can win your main goal while quietly hurting something else — for example, more sign-ups but fewer purchases. This check looks at your other (secondary) goals and warns you if one of them has dropped meaningfully. If it flags a goal, open that goal in the results below and decide whether the main win is still worth it before you ship.

What does 'config changes during the test window' mean?

It means the test was edited after it started — a variant, a goal, or who it targets was changed mid-flight. That mixes together visitors who saw slightly different things, which can quietly spoil the comparison. The card points you to the version history so you can see exactly what changed and when. Small cosmetic edits (fixing a typo, renaming a goal) are usually fine; changing what visitors actually saw usually means you should restart the test.

Why did a blue 'active segment filter' notice appear?

Because you're looking at a slice of your visitors (for example, only mobile, or only one country) instead of everyone. Slices have fewer people and are noisier, so a result that looks strong inside a segment is much more likely to be a coincidence. Clear the filter to read the result for all your traffic before deciding, and treat anything you spot in a segment as an idea to test next — not a conclusion.

Do these checks change or stop my test?

No. The health card only looks and reports — it never pauses, stops, or edits anything on its own. (If you want a test to stop automatically, that's a separate feature: see Stop Conditions.) The checks update by themselves as new visitors and conversions come in, so the card always reflects the latest data.

Analyzing Results5 min read

Experiment Health

Q: Where do I find it?

Open any test and go to its results page. The card is labelled "Experiment health" and starts collapsed, showing a one-line summary like "Looks clean — safe to read" or "A few things worth a look." Click it to open the full list of checks. Click any single check to see why it matters, what your result means right now, and what to do about it.

An automatic checklist on every results page that tells you whether your numbers are safe to trust yet, and what to fix if they aren't.

Browse docs

Every results page has an automatic checklist that tells you whether your numbers are safe to trust yet — and what to fix if they aren’t. Think of it as a friend looking over your shoulder before you call a winner.

A results page includes a card labelled Experiment health. It checks the safeguards that fit that experiment method and gives each one a simple status. Bandits omit both fixed-horizon sample-size and fixed-weight traffic-split checks; secondary-goal, configuration, and segment checks still apply. The card only looks and reports: it never pauses, stops, or edits your test.

The card starts collapsed, showing a one-line headline like “Looks clean — safe to read” or “A few things worth a look.” Click it to open the full list, then click any single check to see why it matters, what your result means right now, and what to do about it. The headline always reflects the most serious thing the card found.

What the statuses mean

All clear

This check looks healthy. Nothing to do.

Warning

Worth a look before you trust the result.

Issue

Stop — something is likely wrong. Fix it before deciding.

Notice

Just information, not a problem.

The checks it runs

Sample size (frequentist manual splits)

Have you collected enough visitors yet?

Frequentist manual splits compare progress with the planned sample size and show roughly how many days remain. Bayesian manual splits have no fixed target, so this check is omitted for them.

Traffic split (manual splits)

Did each version get its fair share of visitors?

Manual splits compare observed traffic with the fixed weights you configured. Bandit weights intentionally change as results arrive, so a fixed-weight sample-ratio check would be misleading and is omitted.

Secondary goals

Is a winner quietly hurting your other goals?

A version can win your main goal while harming something else — more sign-ups but fewer purchases, for example. This watches your other goals and warns you if one of them drops meaningfully, so you can decide whether the main win is still worth it.

Mid-test changes

Was the test edited after it started?

Editing a variant, goal, or targeting after a test starts mixes together visitors who saw different things, which can spoil the comparison. This spots edits made during the test window and points you to the version history so you can see exactly what changed.

Active segment filter

Are you looking at everyone, or just a slice?

Appears only when you’ve filtered the results to a slice of visitors (like mobile only, or one country). Slices are smaller and noisier, so a result that looks strong inside one is more likely to be a coincidence. Clear the filter to read the result for all your traffic before deciding.

Getting the most from it

Open it before you call a winner. A glance at the headline tells you whether to trust the page. On a manual split, a red traffic-split issue usually means fixing the cause and starting a fresh test, because the data already collected can’t be rescued.

A clean card isn’t a guarantee of a real winner. It means the common traps it checks for didn’t fire. You still need a score that’s reached your confidence threshold and a result that makes sense for your business. See Reading Results for how to read the score itself.

Let it do the worrying. The checks update by themselves as new visitors and conversions arrive, so the card always reflects the latest data — you don’t have to re-run anything.

Frequently asked questions

Quick answers to the questions teams ask most about this part of Otter.