Developer Reference7 min read

Notifications & Webhooks

Send test events to Slack or your own webhook, and schedule a daily digest. Five event types, idempotent delivery, signed payloads.

Notifications & Webhooks

Send Otter A/B test events to Slack, your own webhook handler, or as a scheduled daily digest. Five event types, idempotent delivery, signed payloads.

Notifications turn test lifecycle moments into messages somewhere your team already looks. A “test crossed significance” ping in #growth-experiments is more actionable than checking the dashboard every morning. A “test started” ping to your own webhook lets you log experiments into the same system you use for feature flags.

The system has two halves: destinations (where messages go — a Slack webhook URL or your own webhook endpoint) and subscriptions (which events on which tests get sent to which destination). One destination can have many subscriptions; one subscription targets one destination.

Destinations

Slack

slack_webhook

Paste an Incoming Webhook URL from Slack. URL must start with https://hooks.slack.com/. Slack treats the URL itself as the secret — payloads are not separately signed.

Generic webhook

generic_webhook

POST to any HTTPS endpoint you control. A signing_secret is generated automatically, and every request carries an X-OtterAB-Signature header (HMAC-SHA256 of the body) so you can verify authenticity on your side.

URL rules enforced at save time

  • HTTPS only. Plain HTTP is rejected.
  • No private or local hosts. URLs that resolve to localhost, 127.0.0.1, 0.0.0.0, ::1, or any RFC 1918 / link-local address are rejected — this prevents webhooks being aimed at the Otter A/B server's internal network.
  • Slack destinations must use a Slack URL. The hostname check above is in addition to the Slack-specific URL pattern.

Event triggers

Five test events. Each subscription targets exactly one event key (or the daily digest).

  • test.started

    A test moved from draft to running.

  • test.paused

    A running test was paused — new assignments stopped.

  • test.resumed

    A paused or completed test was resumed.

  • test.completed

    A test was marked complete — results are frozen.

  • test.decision_threshold_reached

    A variant's score crossed the effective confidence threshold. (Older name: test.significance_reached — still accepted.)

Daily digest

A daily_digest subscription rolls up activity across your subscribed tests and fires once a day at the hour and timezone you set. Schedule fields are required:

  • hour — integer 0–23.
  • timezone — any IANA timezone (e.g. America/New_York, Europe/London).

Scoping a subscription to specific tests

By default a subscription fires for every test in the account. Set the scope to a list of test IDs and it only fires for those tests. Useful when one Slack channel cares about a single campaign but you don't want every test pinging it.

Delivery lifecycle & idempotency

Each delivery is keyed by a subject_key that's unique per (subscription, event). If a delivery already exists for a given subject, the system reuses it — the same event never fires twice. Failed deliveries are retried under the same subject_key.

  • pending

    Created but not yet attempted. A worker will claim it shortly.

  • delivering

    A worker has claimed this delivery and is sending it now. Stale claims (older than 5 min) are auto-reclaimed for retry.

  • delivered

    Delivered successfully. Records the delivered_at timestamp.

  • failed

    Last attempt errored. The attempts counter and last_error message are tracked for debugging; the delivery is retried with the same subject_key.

Webhook handler tips

Verify the signature on every request. Every webhook carries anX-OtterAB-Signatureheader — an HMAC-SHA256 hex digest of the raw request body keyed with your destination's signing_secret. Recompute on your side and compare in constant time; reject any mismatch. Anyone who guesses your URL can send you a payload otherwise.

Respond with 2xx fast. Treat the webhook as a notification, not a job — enqueue real work to a background processor and return 200. Slow responses risk timeouts and unnecessary retries.

Make your handler idempotent. Otter A/B avoids duplicate sends via subject_key, but transient network errors can still cause the same subject to retry. Use the subject_key in your own dedupe logic.

Pause a subscription before launching big tests. If you're about to spin up dozens of tests via the API, toggle the subscription off, finish the rollout, then toggle it back on — otherwise you'll flood Slack with start events.

Frequently asked questions

Quick answers to the questions teams ask most about this part of Otter A/B.