Framer A/B testing: built-in tests, their limits and an alternative

The same-day conversion window in Framer A/B testing

Framer's tests are cookieless. Each visitor gets an anonymous identifier built from their IP address and browser information, and that identifier resets daily. The consequence, which Framer's help center states, is that conversions are only measured within the same day. Framer says a cookie-based system is planned.

For impulse decisions, that window is fine. For considered ones, it isn't. A buyer who reads your pricing page on Monday and books a demo on Wednesday counts as a visitor who didn't convert, and when they return they may land in a different variant. The longer your sales cycle, the more conversions fall outside the window, and the more the test favors variants that win quick, same-session actions.

Otter identifies visitors with a random ID in a first-party cookie that lasts 365 days, so a visitor keeps the same variant and a conversion on a later visit in the same browser still counts. Two limits apply: in opt-in consent mode Otter waits for consent before setting that cookie, and Otter doesn't follow visitors from one device to another.

Framer's other limits: sections, flicker and plans

Framer's built-in testing is page-level. Its help center says Framer doesn't include built-in A/B testing for custom sections, and points to code overrides, code components or marketplace plugins such as GrowthBook for experiments inside a page. It also notes that Framer sites are React single-page apps, so third-party experiment logic runs client-side and can cause a brief visual flash above the fold. Framer doesn't provide direct support for those custom code setups.

Plan gating is the third limit. Convert needs Pro or Enterprise, so a site on Basic has no native testing at all, and a site on Pro pays for analytics events on top of its plan. Five tests is also a cap that a busy marketing team can reach quickly.

How Framer A/B testing works with Otter

Otter installs through Framer's site-wide custom code: open Site settings > Custom code, paste the Otter snippet into the head, publish, and verify. No export, plugin or code component is needed, and the snippet loads on every published page. It hides the page until test settings load, with a 3-second failsafe, which deals with the above-the-fold flash Framer warns about by keeping the original from showing before the variant.

Otter tests a section, an element or a whole page. In the visual editor you click an element on your live Framer site and change its text, link, image URL, style or HTML, insert content, or remove it, and preview at desktop, tablet and mobile widths, where your real breakpoints apply. For a bigger redesign, build a second page in Framer and run a split URL test between the two. Goals can be clicks, pageviews, custom events, GTM dataLayer events, GA4 events or revenue, and there's no cap on the number of tests.

Statistics are your choice per test: frequentist or Bayesian, at an 80, 90, 95 or 99% threshold, with the bar raised automatically when a test has more than one challenger. Starter costs $39/month for one site with unlimited visitors and tests, on top of your Framer plan; see Otter pricing.

Why Framer sites trip up visual editors

Framer's markup is unusual, and generic visual editors struggle with it in three ways. First, Framer often sets pointer-events: none on text, so a click reports the surrounding container rather than the text you meant. Otter's editor looks under the pointer for the element that owns the text, and its Select parent and Select child controls let you correct the selection. It also refuses a text change on a container that holds two or more separate text blocks, because rewriting it would wipe the siblings.

Second, Framer generates class names, so Otter leaves framer- classes out of the selectors it writes. Third, Framer serves one copy of some content per breakpoint and removes the others when the page hydrates, so a selector based on position can match nothing in the served HTML. Otter anchors selectors to stable IDs where it can and applies a single change to every identical copy, of which at most one is shown. When a change should apply to all matching elements, Otter outlines every match before you apply it and warns when a selector matches too many.

Framer's built-in tests or Otter: which to use

Use Framer's native A/B testing if you're already on Pro, test whole pages, and your visitors decide on the day, such as a waitlist signup from a launch post. It lives inside the editor you use every day. Use Otter if your conversions happen over days, if you want to test sections rather than pages, if you need more than five tests or revenue goals, or if your Framer plan doesn't include Convert. We make Otter, so trial it against Framer's add-on on your own traffic. If you also run sites on other builders, the same account covers them; the Webflow A/B testing guide shows one example.

Whichever you choose, plan the test size before you start. Framer sites often launch with modest traffic, and a signup page converting at 4% needs roughly 18,000 visitors per variant to detect a 15% relative lift at 95% significance and 80% power. The sample size calculator gives the number for your page.

Framer A/B testing and SEO

A Framer site is often the company's main marketing site, so tests should follow Google's website testing guidance: don't cloak, use temporary 302 or JavaScript redirects for split URL tests, give a variant page a rel="canonical" tag pointing to the original, and remove test pages once you have a result. Google notes that small changes, such as button text or placement, often have little or no effect on rankings.

Otter's visual tests keep the URL and change the page with JavaScript, and Otter doesn't assign known bots to a variant, so crawlers see the published original. For split URL tests, keep the variant page out of your navigation and delete it after the test.