Bandit or fixed split? How to pick when your site doesn't get much traffic
Every experiment in evolveAB is one of two kinds, and you pick before it starts because the choice is locked afterwards. An A/B/n test splits traffic evenly and keeps it that way. A bandit starts even and then moves traffic toward whichever variant is doing better.
Bandits sound like the obvious choice: why keep sending half your visitors to a loser? They answer a different question, though, and on a site with modest traffic the difference shows up fast.
#Measuring versus earning
A fixed split exists to measure. Every arm gets the same share for the whole test, so at the end you have a fair estimate for each variant, the losers included. When you need to know whether the new pricing copy is 4% better or 20% better, because that number feeds a decision somewhere else, run a fixed split.
A bandit exists to earn while it learns. It wants more conversions during the test and cares much less about pinning down exactly how bad the weak arms are; once a losing variant stops getting traffic you learn little more about it, and since you weren't going to ship it, that's fine.
Fixed split when you need the number, bandit when you need the conversions.
#How the bandit moves traffic
It uses Thompson sampling. For each variant the cloud keeps a Beta posterior over the conversion rate, Beta(1 + conversions, 1 + visitors − conversions) for a yes/no goal, draws from all of them 20,000 times and counts how often each arm comes out on top. That share, P(best), becomes the arm's traffic weight.
Left alone, that rule would overreact, so four limits rein it in:
- Warm-up. Weights stay equal until every arm has 100 visitors, because before that the posteriors are too wide to mean much.
- Floors. Control always keeps at least 10% of traffic, since everything else gets compared against it; every other arm keeps at least 5% (less when there are many arms), which gives a variant with a bad first week room to recover.
- Pace. The cloud recomputes at most hourly and only when new data has arrived, and it publishes new weights only when one moves by 2 percentage points or six hours have passed.
- Sticky visitors. New weights affect new visitors only. Someone who saw variant B last week keeps seeing B.
#The part bandit explainers skip
On a site with a few hundred visitors a week, the warm-up alone takes a while, and the posteriors stay wide for a long time after it. The bandit will shift traffic, slowly, and it will wobble on the way.
That doesn't rule bandits out for small sites, but it changes how you set one up. Give it a fast goal: a bandit only learns from conversions it has already seen, and a goal with a 30-day window gives it almost nothing to work with in the first month. We recommend bandits for goals whose window is 7 days or less, and autopilot requires it.
And don't read the traffic split as a verdict. If variant C has 60% of traffic, the bandit currently thinks C is most likely best, nothing more. The results table still reports P(beat control), the interval and the state, computed exactly as for a fixed split, and the decision belongs there.
#A quick guide
| You want to | Use |
|---|---|
| Know how big the effect is, for every variant | A/B/n test |
| Make a ship-or-don't call you'll defend later | A/B/n test |
| Get more sign-ups during the test itself | Bandit |
| Keep testing new AI challengers on one region, indefinitely | Bandit, with autopilot |
| Test something with a slow goal (churn, 30-day revenue) | A/B/n test, and patience |
Unsure? Start with a fixed split on your most important page. Bandits earn their keep once you're running a steady stream of variants on the same region, which is what autopilot does.