Marketer guide

This guide is for the people who run experiments day to day: marketers, growth and product folks. You don't need to write code. A developer sets Evolve up once (Quickstart); after that you work in the dashboard at app.evolveab.com.

#What a developer sets up, and what you control

Fields marked locked keep the value in the code: nobody can change them in the dashboard, and the AI never sees them. Links and images are never written by the AI either.

Roles. Admins and Editors can create and change experiments. Viewers see everything but can't change it ("View only: ask an admin for Editor access").

#Where things are

The left sidebar of a site:

• Overview: the setup checklist while you get started, then the site at a glance.
• Experiments: every experiment and its status.
• Proposals: AI-suggested variants waiting for your decision.
• Inbox: what Evolve decided or noticed on its own (auto-pauses, guardrail alerts, variants ready to promote, AI notices).
• AI: the brand profile, forbidden claims, approval settings and the chat webhook.
• Regions and Goals: what your developer has set up.
• Change log: every change to the site's configuration, who made it, newest first.

#Create an experiment

1. Go to Regions, find the region and click New experiment (or Experiments → New experiment).
2. Name it. Check the path: the pages where visitors are assigned.
3. Choose the type. It can't change once the experiment starts.
   • A/B/n test: a fixed split. Read the result when it's decided.
   • Bandit: shifts traffic toward the better variant as data arrives (recomputed hourly). Choose this if you want autopilot.
4. Pick the Primary goal, the one the experiment is decided on. Add secondary goals to watch as well.
5. Optional: add guardrails (below) and lower the share of visitors in the experiment. The rest see control and aren't counted.
6. Edit Variant B: change the fields you want to test. You can have up to 6 variants including control. Control is always what's in the code.
7. Click Save draft. On the experiment page, use Preview to see each variant in your browser (nothing is counted), then click Start.

Each visitor keeps the variant they first got, on every visit. Search engines and other bots always see control.

A started variant can't change, because visitors already saw it. To change one, pause it and add a new variant: Edit as new variant sets that up for you.

#Read the results

Results appear a few minutes after the start. They update on a schedule; Recompute asks for fresh numbers now (at most once a minute). The page shows when the data was last aggregated.

For each goal and variant the Goals table shows:

The State reads:

• Not enough data: too few visitors or conversions to compare. It estimates how many more visitors per variant it needs.
• Deciding: enough data to compare, no decision yet. Keep it running.
• Winner: P(beat control) is at least 99% and the risk of picking it is 1% or less.
• Loser: P(beat control) is 1% or less.

Lift reads "—" until both arms have enough visitors and control has some conversions. "Window not elapsed for X% of visitors" means some visitors still have time to convert, so recent numbers can still go up. That's normal.

The statistics are Bayesian, so the chances above read the way they sound: "P(beat control) 97%" means a 97% chance this variant is better.

The page also shows the traffic split (what share of visitors each variant gets, and its history), the conversion rate over time, and Vitals: page speed (LCP, CLS, INP) per variant, measured in visitors' browsers.

#Guardrails

A guardrail is a goal a variant must not hurt, such as "purchases may get at most 5% worse" while you test sign-ups. Click Add guardrails (or Edit guardrails) on the experiment and set the limit per goal.

• Each variant is compared with control on visitors whose goal window has passed, so a guardrail can show pending for a while ("Waiting for mature data").
• Status per variant: pass, pending or fail.
• A variant can't be promoted until every guardrail passes. With autopilot on, a variant that fails one is paused.
• You can change guardrails at any time, even while the experiment runs. They only change Evolve's decisions, never what visitors see.

Auto-rollback (on by default) also watches page speed: a variant that's clearly slower than control (LCP 20% slower, CLS 0.05 higher or INP 50 ms slower, on enough page views) is paused automatically, and its visitors get control. You'll see it in the Inbox and on the variant ("Paused · slower vitals").

#Finish an experiment

When a variant wins and passes every guardrail, the experiment shows Ready to promote. Click Promote & complete, check the winner and click Complete with winner. From then on everyone sees the winner.

Then the winner goes into the code. With GitHub connected (Growth, Enterprise and the trial), click Create PR on the Promote winner card: Evolve opens a pull request for your developer to review and merge, and the experiment archives itself once the change is deployed. See Winner pull requests. Without GitHub, ask your developer to put the winning copy into the code, then Archive the experiment.

You can also Pause a running experiment at any time (its visitors see control) and Resume it later, or pause a single variant.

#AI proposals

Evolve can write challengers for a region's fields. On a running experiment, click Suggest variants (it takes up to a minute). Every suggestion is checked before you see it:

1. It fits the region's fields and limits (the schema).
2. It doesn't touch locked fields.
3. It contains none of your forbidden claims.
4. It isn't a duplicate of an existing variant.
5. A second AI model, the AI checker (the judge), reads it for forbidden or unsupported claims.

Suggestions that pass go to Proposals → Waiting. Each card shows control and the proposal field by field, with the changes highlighted, why the AI thinks it might win, and the checks it passed. Suggestions that failed a check are listed under Filtered, with the reason.

For each proposal:

• Approve: it becomes a variant and goes live at no more than 20% of the experiment's traffic until it has enough data.
• Edit, then approve: change the text first. Your edit goes through the same checks again.
• Reject: say why (required). The next suggestions learn from it.

Nothing the AI writes goes live without your approval, unless you turn on autopilot.

#Brand profile and forbidden claims

On the AI page:

• Brand profile: your voice, audience and the facts your site states. Every suggestion is written and checked against it. Click Generate from my site to draft one from your pages, review it, then Save brand profile. Each save is a new version, and you can revert.
• Forbidden claims: words or phrases suggestions must never contain (for example "guaranteed" or "#1"). Evolve ships defaults you can turn off, and you can add your own. The AI checker also catches paraphrases.
• AI suggestions and Judge every suggestion switches. Turning AI suggestions off stops new suggestions; serving and results are unaffected.

#Autopilot

Autopilot runs a bandit experiment for you. With it on, Evolve:

• retires clear losers (they stop getting traffic: "Retired · clear loser"),
• writes new AI variants to keep the number of variants you set (Target arms),
• approves only suggestions that pass every check and the AI checker. New variants start at no more than 20% of traffic.

Rollback keeps running either way. When all 6 variant slots have been used, Start next round creates a new draft with control and the surviving variants.

Autopilot can be turned on (Turn on autopilot, on the experiment's Autopilot card) when all of these hold. The card shows each one:

• the experiment is a running bandit,
• its primary goal's window is 7 days or less,
• auto-rollback is on,
• AI is allowed for the site,
• aggregates are fresh and aggregation is healthy,
• no guardrail is failing,
• a person has saved the brand profile.

If the results stop updating, autopilot pauses itself: no variants are retired, added or approved until the data is fresh again.

#Notifications

The Inbox lists what Evolve did or noticed on its own: auto-pauses (including for slow vitals), guardrail failures, losing variants, variants ready to promote, autopilot actions and AI errors or limits. To get them in Slack, add a Slack-compatible incoming webhook on the AI page (admins only).

#Good habits

• Test one idea per variant, so you know what made the difference.
• Choose the primary goal before you start, and don't switch to whichever goal looks best afterwards.
• Let a test reach Winner or Loser instead of stopping when it first looks good. Sites with little traffic need more time.
• Add a guardrail on revenue or sign-ups when you test anything near the checkout.
• Write down why you rejected a proposal; it improves the next ones.