Experimentation
Reading results, Bayesian statistics, bandits, guardrails and the habits that keep tests honest.
Bandit or fixed split? How to pick when your site doesn't get much traffic
A fixed A/B split gives you a clean answer. A Thompson-sampling bandit sends more visitors to the better variant while it learns. Which one fits depends on the question you're asking, not on which sounds smarter.
We set our winner bar at 95%. A simulation talked us out of it.
Our first results engine called a winner at P(beat control) ≥ 95%. Run against two identical variants with ten looks, it crowned a false winner roughly 15% of the time. Here's what we changed.
