{noEffect ? `With a true lift of 0 %, every win is a false winner.${oneSided ? " Stopped tests are not results, so they add no false losers." : " Every loss is a false loser."} Set a true lift to see how often a real effect is found.` : "With a real lift, the win share is how often the test finds it before its maximum runtime."}{" "}
The simulation assumes the same traffic every day and one variation. Real traffic changes by weekday and season, so treat the numbers as a guide.
On day {horizon}, a variation needs a p-value below {needed[needed.length - 1].toFixed(4)}.
;
};
The frequentist engine answers one question: **if the variation really made no difference, how surprising would this data be?** If the data would be very surprising, pagent calls a result.
Choose this method when your team thinks in p-values and confidence levels, or when you need an exact false-winner rate for settings you choose yourself.
## The numbers you see
| Number | What it means in plain words |
| - | - |
| **p-value** | How likely a difference at least this large would be if the variation had no real effect. Lower means stronger evidence. |
| **Significance level** | The p-value a result must get below. pagent adjusts it at each check, see [sequential correction](#sequential-correction). |
| **Confidence interval** | The range of lifts that fit the data. Its level is 1 minus the significance level: 90 % at Balanced. |
## How a result is called
At each [scheduled check](/docs/statistics/overview#when-pagent-checks-a-test), after the [shared rules](/docs/statistics/overview#the-order-of-the-checks) for data and runtime pass:
1. **Equivalence check.** If [practical equivalence](/docs/statistics/settings#practical-equivalence-rope) is on and the variation is clearly within the band around zero, the test stops as **inconclusive**. pagent uses two one-sided tests: the confidence interval at 1 − 2 × the significance level (80 % at Balanced) must lie fully inside the band.
2. **Significance check.** pagent runs a two-proportion z-test on the primary goal. If the p-value is below the threshold for this check:
* and the variation is better, the test stops with the **variation as the winner**,
* and the variation is worse, the test stops as a **Loss**.
3. Otherwise the test keeps running.
By default the test is **two-sided**: the significance level is split between a false win and a false loss. At Balanced (10 %), about 1 in 20 tests where nothing changed ends as a false winner and 1 in 20 as a false loser.
### One-sided frequentist tests
Under **Advanced settings → Frequentist**, you can set the [decision policy](/docs/statistics/settings#decision-policy) to **One-sided**. Then:
* the whole significance level goes to the win side, so 10 % means about 1 in 10 false winners,
* the p-value shown is one-sided,
* a variation that is significantly worse still stops the test, as **Stopped: variation was losing**, without counting as a result.
The levels always use two-sided frequentist tests. A one-sided policy shows as **Custom settings**.
## Settings
The frequentist engine has three settings of its own. All [shared settings](/docs/statistics/settings#settings-for-both-engines) apply as well.
### Significance level
The accepted chance of a false result when the variation makes no difference. Default **10 %** (Balanced), allowed above 0 % and up to 50 %.
Each level sets the significance level to twice its promise, because two-sided tests split it:
| Level | Significance level | False winners at most | False losers at most |
| - | - | - | - |
| Explore | 30 % | 1 in 7 | 1 in 7 |
| Fast | 20 % | 1 in 10 | 1 in 10 |
| Balanced | 10 % | 1 in 20 | 1 in 20 |
| Careful | 5 % | 1 in 40 | 1 in 40 |
| Strict | 2 % | 1 in 100 | 1 in 100 |
A classic "95 % confidence, two-sided" standard is the **Careful** level. A lower level means fewer false results, but tests need more data to call a real effect.
### Sequential correction
**On by default** and in every level. We recommend keeping it on.
pagent looks at a running test many times, once per check. Each look is another chance for random noise to cross the line. Without a correction, a test checked every day for three weeks calls far more false results than the significance level promises.
The sequential correction spreads the significance level over the life of the test. It uses O'Brien-Fleming alpha spending:
* Early checks need very strong evidence. Only large, clear effects stop a test early.
* The bar drops as data comes in.
* The final check, at the [maximum runtime](/docs/statistics/settings#maximum-runtime-days), sits close to the plain significance level.
See the p-value a variation needs at each daily check:
pagent measures progress by visitors. It estimates how many visitors the test will collect by its maximum runtime from the traffic of the first check. To keep a decision possible at every check, the progress never runs ahead of the share of runtime that has passed.
With the correction off, every check uses the plain threshold, and false results are several times more frequent than the significance level. Only turn it off if you look at the result once, at a fixed end date. In that case, also turn [automatic stopping](/docs/statistics/settings#automatic-stopping) off and stop the test yourself on that date.
### Decision policy
Two-sided by default, one-sided as an option. See [One-sided frequentist tests](#one-sided-frequentist-tests).
## How often a false winner is called
Results of 20,000 simulated tests per cell with each level's frequentist settings, a 3 % conversion rate and one check a day:
| Level (promise) | False winners at 300 / 3,000 / 30,000 visitors a day | Real +5 % lift found | Real +10 % lift found |
| - | - | - | - |
| Explore (1 in 7) | 11.8 % / 2.2 % / 0.0 % | 21 % / 12 % / 0 % | 33 % / 38 % / 2 % |
| Fast (1 in 10) | 9.7 % / 6.4 % / 0.0 % | 17 % / 31 % / 3 % | 28 % / 69 % / 80 % |
| Balanced (1 in 20) | 4.8 % / 4.4 % / 0.1 % | 12 % / 39 % / 46 % | 23 % / 86 % / 100 % |
| Careful (1 in 40) | 2.6 % / 2.7 % / 0.3 % | 8 % / 41 % / 88 % | 18 % / 93 % / 100 % |
| Strict (1 in 100) | 1.0 % / 1.0 % / 0.9 % | 4 % / 34 % / 100 % | 13 % / 94 % / 100 % |
False losers are about as frequent as false winners. At high traffic, the wide no-difference bands of Explore and Fast end most tests early as "no meaningful difference", including tests with a real lift smaller than the band.
Try your own scenario. Pick a level, set the true lift to 0 % to count false results, or to a real lift to see how often it is found and how fast:
## What the frequentist engine does not do
* It does not give a "chance to beat control". A p-value is not the probability that the variation is better.
* It has no expected-loss check. That is a Bayesian setting.
* It has no power setting. pagent does not plan a sample size up front; the [maximum runtime](/docs/statistics/settings#maximum-runtime-days) sets the end.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.