Which test type should I pick for my experiment?

Use a Split URL test when each variant has its own URL (Shopify theme swaps, alternate landing pages, a/b versions you've already deployed). Use an A/B/n Visual test when you want to change something on the same page — headlines, CTAs, layouts, images. Use Personalization when there's no winner to discover, only different content for different audiences (geo banners, returning visitor messaging, UTM variants).

How many variants can I run at once?

There's no hard cap, but for statistical reasons we recommend two or three challenger variants max for A/B/n tests. Every extra variant requires more traffic to reach significance because the Bonferroni correction raises the bar each time. Five-variant tests need roughly twice the traffic of a two-variant test.

Can I edit a test after it's started?

You can change non-statistical settings (name, collaborators, description) anytime. Changing variants, goals, or targeting after launch is allowed but invalidates the data collected so far — the results page will warn you and offer to reset. The wizard explicitly highlights the risky edits during configuration.

What's the traffic split between variants?

A manual split starts evenly and lets you set fixed weights in the Variants step — useful for a risky variant or 90/10 holdout. A multi-armed bandit also starts evenly, then adapts new-visitor weights from primary-goal results. Either method keeps each visitor on their first assigned version.

Do I need to set up goals before launching?

Yes. Every test needs at least one goal, with exactly one marked primary. The primary goal drives the win/lose decision; secondaries are tracked for reference. The Goals page covers the six goal types in detail.

What does the sample-size advisor do?

For a frequentist manual split, the Review step estimates how long the test needs to run from your traffic, baseline conversion rate, and minimum detectable effect. It is guidance, not a hard stop. Bayesian manual splits do not show a fixed-horizon estimate. Bandits also omit it because they adapt traffic continuously.

Can I save a test as a draft and come back?

Yes. The Review step has both a Launch button and a Save as Draft button. Drafts retain all your wizard inputs and can be edited at any time before launch. There's no expiration on drafts.

Building Tests7 min read

Creating a Test

Test types, the six-step wizard, and how variants are configured.

Browse docs

Every test in Otter is built through the same six-step wizard. The flow starts by picking a test type, which decides what variants look like and which downstream questions you get asked.

A good A/B test starts well before you open the wizard. Decide what hypothesis you're testing (“a clearer CTA increases signups”), how big a change you'd need to see to act on it (the minimum detectable effect), and which page or flow you want to run it on. The wizard handles the mechanics; you bring the question.

Otter supports four test types: whole-page swaps via redirect, in-page DOM changes through the visual editor, audience-based personalization, and whole-theme WordPress tests through the official plugin. Pick the one that matches what you're trying to do, not the one that sounds most powerful.

Test types

WordPress Theme Test

Official WordPress Plugin

Splits visitors between the site's current theme and one or more installed alternate themes. The plugin selects the assigned theme for each request without changing the active theme in WordPress, and can open a prefilled site-wide test wizard.

Best for: Comparing a redesign, block theme, classic theme, or child theme across the whole WordPress site.

Split URL / Redirect Test

Redirect

Each variant has its own destination URL. The SDK redirects assigned visitors to the variant URL on page load (with a loop guard to prevent bouncing). Good for testing entire pages, themes, or flows that already live at different URLs.

Best for: Whole-page or whole-flow comparisons, Shopify theme previews, server-rendered alternates.

A/B/n Test

Visual Editor

Variants are sets of DOM mutations applied to the same page — text swaps, style overrides, element hide/show. You build them in the visual editor or by adding raw changes by hand. Results are scored statistically.

Best for: Headlines, CTAs, images, copy changes, layout tweaks — anything you can express as DOM changes without a code deploy.

Content Personalization

Visual Editor

Uses the same visual editor as A/B/n, but with targeting instead of randomization. Each variant has its own audience rules, and matching visitors always see that variant. Useful when there's no "winner" to discover — you just want different content for different audiences.

Best for: Geo-specific banners, returning-vs-new visitor messaging, UTM-driven landing variations.

Wizard steps

1
Type
Choose Split URL / Redirect, A/B/n (Visual), Content Personalization, or WordPress Theme Test. WordPress Theme Test appears for projects using the WordPress or WooCommerce installation guide.
2
Basics
Name the test, set the page URL it should run on, and pick a URL match type (exact, simple, contains, wildcard, regex — see URL Matching). Optionally assign teammates as collaborators.
3
Variants
Define each variant (Control + one or more challengers). Manual splits use the weights you configure; multi-armed bandits start evenly and adapt new-visitor traffic from primary-goal results. Redirect tests use a destination URL, visual / personalization tests use DOM changes, and WordPress theme tests use the installed theme's stylesheet identifier.
4
Goals
Add one or more goals and mark exactly one as primary. The primary goal drives the win/lose decision; secondaries are reported alongside but do not affect the score. See the Goals section for details on the six goal types.
5
Targeting
Filter who sees the test using a rules tree (AND/OR groups) over fields like country, device, traffic source, UTM tags, time of day, custom properties, cookies, and URL parameters. Empty targeting means "everyone matching the page URL".
6
Review
Confirm the full configuration. For manual frequentist splits, the sample-size advisor estimates how long the test needs to run for your traffic and effect size. Bayesian manual splits do not show a fixed-horizon estimate. Bandits use adaptive traffic and Bayesian reporting, so they do not show one either. Launch immediately or save as a draft.

Personalization tests swap the order of Goals and Targeting — targeting comes first because it defines the variant audience, and goals are tracked for reference rather than driving a winner.

WordPress cache requirement

Exclude the otter_theme_assignment cookie from full-page cache, or bypass full-page caching while a WordPress theme test runs. Object caches are fine. A page cache that ignores this cookie can serve one visitor's rendered theme to another.

Keep the active and alternate themes unchanged while the experiment runs. Complete or pause the test before deploying theme changes so each variant remains statistically comparable.

Before you launch

Have a falsifiable hypothesis. “Make the page better” is not a hypothesis. “Adding social proof above the CTA increases signups by ≥5%” is. You can't fail a test you can't define.

Pick a primary goal that maps to revenue. Clicks are easy to win and don't always carry through to purchases. Whenever possible, set your primary goal downstream of the change you're testing.

Respect the sample size estimate on frequentist manual splits. The wizard shows roughly how long that fixed split needs to run for your traffic and effect size. Bayesian manual splits and bandits do not show a fixed-horizon estimate; bandit allocation also adapts as primary-goal results arrive.

Frequently asked questions

Quick answers to the questions teams ask most about this part of Otter.