# Creating a Test Test types, the six-step wizard, and how variants are configured. Every test in Otter is built through the same six-step wizard. The flow starts by picking a test type, which decides what variants look like and which downstream questions you get asked. A good A/B test starts well before you open the wizard. Decide what hypothesis you're testing ("a clearer CTA increases signups"), how big a change you'd need to see to act on it (the minimum detectable effect), and which page or flow you want to run it on. The wizard handles the mechanics; you bring the question. The three test types below cover almost every experiment people run on a website: whole-page swaps via redirect, in-page DOM changes via the visual editor, and audience-based personalization where there's no winner — just different content for different people. Pick the one that matches what you're trying to do, not the one that sounds most powerful. ## Test types ### Split URL / Redirect Test (`redirect`) *Redirect* Each variant has its own destination URL. The SDK redirects assigned visitors to the variant URL on page load (with a loop guard to prevent bouncing). Good for testing entire pages, themes, or flows that already live at different URLs. **Best for:** Whole-page or whole-flow comparisons, Shopify theme previews, server-rendered alternates. ### A/B/n Test (`visual`) *Visual Editor* Variants are sets of DOM mutations applied to the same page — text swaps, style overrides, element hide/show. You build them in the visual editor or by adding raw changes by hand. Results are scored statistically. **Best for:** Headlines, CTAs, images, copy changes, layout tweaks — anything you can express as DOM changes without a code deploy. ### Content Personalization (`personalization`) *Visual Editor* Uses the same visual editor as A/B/n, but with targeting instead of randomization. Each variant has its own audience rules, and matching visitors always see that variant. Useful when there's no "winner" to discover — you just want different content for different audiences. **Best for:** Geo-specific banners, returning-vs-new visitor messaging, UTM-driven landing variations. ## Wizard steps 1. **Type** — Choose Split URL / Redirect, A/B/n (Visual), or Content Personalization. This decides which downstream steps appear and how variants are configured. 2. **Basics** — Name the test, set the page URL it should run on, and pick a URL match type (exact, simple, contains, wildcard, regex — see URL Matching). Optionally assign teammates as collaborators. 3. **Variants** — Define each variant (Control + one or more challengers) and the traffic split between them. For redirect tests you provide a destination URL per variant. For visual / personalization tests you provide DOM changes — typically authored in the visual editor. 4. **Goals** — Add one or more goals and mark exactly one as primary. The primary goal drives the win/lose decision; secondaries are reported alongside but do not affect the score. See the Goals section for details on the five goal types. 5. **Targeting** — Filter who sees the test using a rules tree (AND/OR groups) over fields like country, device, traffic source, UTM tags, time of day, custom properties, cookies, and URL parameters. Empty targeting means "everyone matching the page URL". 6. **Review** — Confirm the full configuration. The sample-size advisor estimates how long the test needs to run for your traffic and effect size. Launch immediately or save as a draft. > **Note:** Personalization tests swap the order of **Goals** and **Targeting** — targeting comes first because it defines the variant audience, and goals are tracked for reference rather than driving a winner. > **Before you launch:** > > - **Have a falsifiable hypothesis.** "Make the page better" is not a hypothesis. "Adding social proof above the CTA increases signups by ≥5%" is. You can't fail a test you can't define. > - **Pick a primary goal that maps to revenue.** Clicks are easy to win and don't always carry through to purchases. Whenever possible, set your primary goal downstream of the change you're testing. > - **Respect the sample size estimate.** The wizard shows roughly how long the test needs to run for your traffic and effect size. Stopping a test the moment it crosses significance is one of the easiest ways to ship a false positive. ## Frequently asked questions ### Which test type should I pick for my experiment? Use a Split URL test when each variant has its own URL (Shopify theme swaps, alternate landing pages, a/b versions you've already deployed). Use an A/B/n Visual test when you want to change something on the same page — headlines, CTAs, layouts, images. Use Personalization when there's no winner to discover, only different content for different audiences (geo banners, returning visitor messaging, UTM variants). ### How many variants can I run at once? There's no hard cap, but for statistical reasons we recommend two or three challenger variants max for A/B/n tests. Every extra variant requires more traffic to reach significance because the Bonferroni correction raises the bar each time. Five-variant tests need roughly twice the traffic of a two-variant test. ### Can I edit a test after it's started? You can change non-statistical settings (name, collaborators, description) anytime. Changing variants, goals, or targeting after launch is allowed but invalidates the data collected so far — the results page will warn you and offer to reset. The wizard explicitly highlights the risky edits during configuration. ### What's the traffic split between variants? By default, traffic is split evenly across all variants (50/50 for two variants, 33/33/33 for three, and so on). You can override this in the Variants step — useful when you want to limit exposure to a risky variant or run a 90/10 holdout. The split is enforced via deterministic hashing, so the same visitor always lands on the same variant. ### Do I need to set up goals before launching? Yes. Every test needs at least one goal, with exactly one marked primary. The primary goal drives the win/lose decision; secondaries are tracked for reference. The Goals page covers the five goal types in detail. ### What does the sample-size advisor do? On the Review step, Otter estimates how long the test needs to run based on your traffic, baseline conversion rate, and the minimum detectable effect you care about. It's an estimate, not a hard stop — use it to set expectations and avoid running underpowered tests. ### Can I save a test as a draft and come back? Yes. The Review step has both a Launch button and a Save as Draft button. Drafts retain all your wizard inputs and can be edited at any time before launch. There's no expiration on drafts. --- Canonical page: https://www.otterab.com/docs/building-tests/creating-a-test