A/B test significance calculator
Is B really better, and what does it cost if you're wrong? Bayesian, and it all runs in your browser.
Example result
- Chance B beats A
- 92%
- Expected loss if you ship B
- 0.0355 points
- Expected loss if you keep A
- 1.5 points
B converts at 6.5% against A's 5%, 30% higher in relative terms so far.
Three numbers, in plain words
- Chance B beats A
- How likely it is that B's true conversion rate is higher than A's, given what you've seen.
- Expected loss if you ship B
- If A is actually better, how much conversion rate you'd give up by shipping B, averaged over every way that could be true, in percentage points of conversion rate. Small means a wrong call is cheap.
- Expected loss if you keep A
- The same, the other way round: what staying with A costs if B is the better one.
Watch the result come into focus
Example numbers, replayed Both arms fill up at the conversion rates above, from a handful of visitors to the full count. It shows how sure the model can be at each size. It isn't your test's real day-by-day history.
1,000 visitors on A, 1,000 on B
Chance B beats A92%
How it's worked out
A Beta-Bernoulli model, the same kind evolveAB uses for its experiment results.
- The prior
- Each arm's true rate is unknown, so the model holds a belief about it: a Beta distribution. It starts flat, Beta(1, 1), where every rate from 0% to 100% is equally likely.
- The posterior
- Each visitor converts or doesn't. After c conversions from n visitors, the belief is Beta(1 + c, 1 + n − c). Those are the curves in the chart.
- Chance B beats A
- P(B > A) = ∫ fB(x) FA(x) dx: B's curve, weighted by how much of A's curve sits below each point. Worked out on a 6,000-point grid, in your browser.
- Expected loss
- For shipping B it's E[max(A − B, 0)], the average shortfall counting only the cases where A is better. Keeping A is the mirror image. Both are in percentage points of conversion rate.
Worth knowing
- There's no p-value here. The output is a probability about the two rates, which is what most people want a p-value to be.
- Small samples give wide curves. A probability from 40 visitors can swing a long way with the next 40.
- Planning a test? Work out how many visitors it needs first, with the sample size calculator.
Ship B or keep A?
B Ship B
92%
chance it's the better one. If it isn't, you lose 0.0355 points of conversion rate on average.
A Keep A
8%
chance it's the better one. If it isn't, you give up 1.5 points on average.
Not enough traffic for a fixed test? A bandit moves traffic to the leader as it learns. How bandits work in evolveAB