{noEffect ? `With a true lift of 0 %, every win is a false winner.${oneSided ? " Stopped tests are not results, so they add no false losers." : " Every loss is a false loser."} Set a true lift to see how often a real effect is found.` : "With a real lift, the win share is how often the test finds it before its maximum runtime."}{" "}
The simulation assumes the same traffic every day and one variation. Real traffic changes by weekday and season, so treat the numbers as a guide.
A check only decides once the test has the minimum conversions and runtime. This calculator uses a close approximation; the
results page shows pagent's exact values.
;
};
The Bayesian engine is pagent's default. It answers the question most people actually ask: **how likely is it that the variation is better than the original?**
## The numbers you see
| Number | What it means in plain words |
| - | - |
| **Chance to beat control** | How likely it is, given the data so far, that the variation converts better than control. 94 % means: in 94 of 100 cases like this, the variation really is better. |
| **Lift** | How much better or worse the variation converts, relative to control. +8 % at a 3 % conversion rate means about 3.24 %. |
| **95 % credible interval** | The range the true lift lies in with 95 % probability. A narrow range means pagent is sure about the size of the effect. |
| **Expected loss** | How much conversion rate you would lose on average if you shipped the variation and it was in fact worse. Measured in percentage points (pp). |
## How a result is called
At each [scheduled check](/docs/statistics/overview#when-pagent-checks-a-test), after the [shared rules](/docs/statistics/overview#the-order-of-the-checks) for data and runtime pass. The numbers are the Balanced level:
1. **Practical equivalence.** If [ROPE](/docs/statistics/settings#practical-equivalence-rope) is on and there is at least a 95 % chance that the true lift is inside the band (±7.5 %), the test stops as **inconclusive**.
2. **Winner.** If the chance to beat control reaches the [threshold](/docs/statistics/settings#chance-to-beat-control-threshold) (98.5 %), the variation **wins**.
3. **Losing variation.** If the chance falls to the [stop losing threshold](/docs/statistics/settings#stop-losing-variations-at) (5 %) or below, the test stops as **Stopped: variation was losing**. This protects your conversions but is not counted as a result.
4. **Expected loss (Careful and Strict).** If the [expected loss check](/docs/statistics/settings#enable-expected-loss-threshold) is on, the test only stops when the expected loss of the leading side is below the [expected loss threshold](/docs/statistics/settings#expected-loss-threshold).
5. Otherwise the test keeps running.
This is the **one-sided** [decision policy](/docs/statistics/settings#decision-policy), the Bayesian default. Only a winner is a statistical result. Under the **two-sided** policy, a variation whose chance falls to 100 % minus the threshold (1.5 % at Balanced) is a **Loss**, and both directions count as results.
Try it with your own numbers. The calculator shows what a single check would decide once the minimum data and runtime are met:
## Priors
Before a test has data, pagent has to start from some assumption. That starting point is the **prior**.
### Skeptical prior (default)
Most changes to a website move conversion by a few percent at most. A +40 % lift on day two is almost always noise. The skeptical prior builds this in: it starts from "this change probably does little" and pulls early, extreme lifts toward zero.
* With little data, the pull is strong. A big early jump counts for less.
* With more data, the pull fades. The data decides.
The [prior width](/docs/statistics/settings#prior-width) (30 % in every level) sets how strong the pull is. At 30 %, pagent expects about two thirds of true lifts to fall between −30 % and +30 %. A smaller width pulls harder.
See how the lift pagent works with approaches the raw lift as visitors grow:
### Flat prior
The flat prior assumes nothing. Every conversion rate between 0 % and 100 % is equally likely before the test starts, so the chance to beat control follows the raw data from the first visitor on.
Websites that existed before October 2026 were kept on the flat prior when the skeptical prior was introduced. Choosing a level switches them to the skeptical prior.
## How often a false winner is called
pagent checks a running test many times, by default once a day. Every check is another chance for noise to cross the threshold, so a single threshold does not tell you how often a whole test ends with a false winner. That is why the levels use high thresholds: each was set with simulations of tests where the variation does nothing, checked daily, at 100 to 50,000 visitors per variation per day, so that even the worst traffic stays within the level's promise.
Results of 20,000 simulated tests per cell with each level's Bayesian settings, a 3 % conversion rate and one check a day:
| Level (promise) | False winners at 300 / 3,000 / 30,000 visitors a day | Real +5 % lift found | Real +10 % lift found |
| - | - | - | - |
| Explore (1 in 7) | 5.1 % / 8.2 % / 0.0 % | 11 % / 33 % / 0 % | 19 % / 69 % / 16 % |
| Fast (1 in 10) | 1.5 % / 7.4 % / 0.0 % | 4 % / 39 % / 14 % | 9 % / 81 % / 95 % |
| Balanced (1 in 20) | 0.6 % / 3.7 % / 0.7 % | 2 % / 34 % / 60 % | 6 % / 84 % / 100 % |
| Careful (1 in 40) | 0.1 % / 1.5 % / 1.1 % | 1 % / 27 % / 89 % | 3 % / 85 % / 100 % |
| Strict (1 in 100) | 0.1 % / 0.7 % / 0.9 % | 1 % / 25 % / 100 % | 3 % / 89 % / 100 % |
Read the found columns left to right as 300, 3,000 and 30,000 visitors per variation per day.
* **Every level keeps its promise** at every traffic level.
* **Low traffic finds little.** At 300 visitors a day, a 5 % lift is rarely found within the runtime at any level. Test bolder changes, or pages with more traffic.
* **Fast levels skip small lifts at high traffic.** Their wide no-difference band (±15 % for Explore, ±10 % for Fast) ends tests with a real but small lift as "no meaningful difference". If lifts around 5 % matter, use Balanced or stricter.
* **About 1 in 7 tests that change nothing stop as losing** at Balanced and 3,000 visitors a day. These are not results, so they add no false losers.
Run your own scenario. Pick a level, set the true lift to 0 % to count false winners, or to a real lift to see how often it is found and how fast:
## Expected loss
The chance to beat control tells you how likely the variation is better. It does not tell you how much is at stake if it is not. Expected loss covers that: it is the average conversion rate you give up by shipping the variation, counting only the cases where it is actually worse.
Example: control converts at 3.0 %. An expected loss of 0.02 pp means that shipping the variation costs, on average, 3.0 % → 2.98 % in the unlucky cases. That is a small risk.
At Explore, Fast and Balanced, expected loss is shown but does not stop tests. Careful and Strict require it to be small, 0.1 pp and 0.05 pp, before a test stops. You can turn the check on for any setup with [Enable expected loss threshold](/docs/statistics/settings#enable-expected-loss-threshold).
## Sequential correction (deprecated)
Older websites may still have the Bayesian **Sequential correction** on. It raises the threshold at early checks, following the same O'Brien-Fleming schedule as the [frequentist correction](/docs/statistics/frequentist#sequential-correction), and the results page then shows the raised threshold.
It is being retired. You can switch it off, but not back on, and tests are two-sided while it is on. Choosing a level switches it off.
## What the Bayesian engine does not do
* It has no p-value. The chance to beat control is not 1 minus a p-value.
* It has no exact false-winner rate for a custom threshold. The levels' promises come from simulation; for an exact rate, use the [frequentist engine](/docs/statistics/frequentist).
* It has no power setting. The [maximum runtime](/docs/statistics/settings#maximum-runtime-days) sets the end of a test.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.