Creating a Test
Test types, the six-step wizard, and how variants are configured.
Creating a Test
Every test in Otter A/B is built through the same six-step wizard. The flow starts by picking a test type, which decides what variants look like and which downstream questions you get asked.
A good A/B test starts well before you open the wizard. Decide what hypothesis you're testing (“a clearer CTA increases signups”), how big a change you'd need to see to act on it (the minimum detectable effect), and which page or flow you want to run it on. The wizard handles the mechanics; you bring the question.
The three test types below cover almost every experiment people run on a website: whole- page swaps via redirect, in-page DOM changes via the visual editor, and audience-based personalization where there's no winner — just different content for different people. Pick the one that matches what you're trying to do, not the one that sounds most powerful.
Test types
Split URL / Redirect Test
RedirectEach variant has its own destination URL. The SDK redirects assigned visitors to the variant URL on page load (with a loop guard to prevent bouncing). Good for testing entire pages, themes, or flows that already live at different URLs.
Best for: Whole-page or whole-flow comparisons, Shopify theme previews, server-rendered alternates.
A/B/n Test
Visual EditorVariants are sets of DOM mutations applied to the same page — text swaps, style overrides, element hide/show. You build them in the visual editor or by adding raw changes by hand. Results are scored statistically.
Best for: Headlines, CTAs, images, copy changes, layout tweaks — anything you can express as DOM changes without a code deploy.
Content Personalization
Visual EditorUses the same visual editor as A/B/n, but with targeting instead of randomization. Each variant has its own audience rules, and matching visitors always see that variant. Useful when there's no "winner" to discover — you just want different content for different audiences.
Best for: Geo-specific banners, returning-vs-new visitor messaging, UTM-driven landing variations.
Wizard steps
- 1
Type
Choose Split URL / Redirect, A/B/n (Visual), or Content Personalization. This decides which downstream steps appear and how variants are configured.
- 2
Basics
Name the test, set the page URL it should run on, and pick a URL match type (exact, simple, contains, wildcard, regex — see URL Matching). Optionally assign teammates as collaborators.
- 3
Variants
Define each variant (Control + one or more challengers) and the traffic split between them. For redirect tests you provide a destination URL per variant. For visual / personalization tests you provide DOM changes — typically authored in the visual editor.
- 4
Goals
Add one or more goals and mark exactly one as primary. The primary goal drives the win/lose decision; secondaries are reported alongside but do not affect the score. See the Goals section for details on the five goal types.
- 5
Targeting
Filter who sees the test using a rules tree (AND/OR groups) over fields like country, device, traffic source, UTM tags, time of day, custom properties, cookies, and URL parameters. Empty targeting means "everyone matching the page URL".
- 6
Review
Confirm the full configuration. The sample-size advisor estimates how long the test needs to run for your traffic and effect size. Launch immediately or save as a draft.
Personalization tests swap the order of Goals and Targeting — targeting comes first because it defines the variant audience, and goals are tracked for reference rather than driving a winner.
Before you launch
Have a falsifiable hypothesis. “Make the page better” is not a hypothesis. “Adding social proof above the CTA increases signups by ≥5%” is. You can't fail a test you can't define.
Pick a primary goal that maps to revenue. Clicks are easy to win and don't always carry through to purchases. Whenever possible, set your primary goal downstream of the change you're testing.
Respect the sample size estimate. The wizard shows roughly how long the test needs to run for your traffic and effect size. Stopping a test the moment it crosses significance is one of the easiest ways to ship a false positive.
Frequently asked questions
Quick answers to the questions teams ask most about this part of Otter A/B.