Skip to main content
Every statistics setup trades speed against certainty. A looser bar ends tests sooner and finds more winners, but more of them are luck. A stricter bar calls fewer false winners, but tests need more visitors. pagent packs this trade-off into five levels. Pick one under Settings → Tests → Statistics.

​
Pick a level

A few rules of thumb:
  • Small lifts need a strict level. Explore and Fast treat lifts below ±15 % and ±10 % as “no meaningful difference”. If a 5 % lift would matter to you, use Balanced or stricter.
  • Low traffic needs patience, not a looser level. Below a few hundred visitors per variation per day, few tests find a 5 % lift at any level. Test bolder changes, or longer: raise the maximum runtime under Advanced settings.
  • Pick the method by how you read results. Bayesian shows a chance to win; frequentist shows p-values. Each level keeps the same promise with both.
Statistics settings lists the exact values each level sets.

​
Match your company’s standard

Many teams have a rule like “we test at 95 % confidence”. Enter yours to find the level that keeps it: The common cases:

​
One-sided and two-sided standards

A two-sided standard checks for wins and losses with the same bar. “95 % confidence, two-sided” allows 2.5 % false winners and 2.5 % false losers. A one-sided standard only checks for wins. “95 % confidence, one-sided” allows 5 % false winners. pagent’s Bayesian levels are one-sided: only a win counts as a result, and a clearly losing variation stops the test without being recorded as a loss. The frequentist levels are two-sided. Either way, the level promises the share of false winners, so you can compare it directly with your standard. See Decision policy.
A chance to beat control is not the same as “confidence”. A Bayesian threshold of 95 % with daily checks calls more false winners than a 95 % confidence standard. The levels take this into account; a custom threshold does not.

​
Several variations

You do not need to adjust anything for the number of variations. pagent tightens the bar for each comparison automatically, for both methods, so the level’s promise holds for the whole test. See Variations and goals.

​
What your traffic can detect

pagent has no power or sample-size setting. To see what your traffic can find, use the simulator with a real lift: the win share is how often a test finds that lift before its maximum runtime. If the share is low, test bolder changes, raise the maximum runtime, or choose a page with more traffic.

​
Changing your setup

  1. Go to Settings → Tests. Under Statistics, choose the method and the level.
  2. Open Advanced settings only if you need a rule the levels do not cover. Changing a value there shows Custom settings.
  3. Click Save changes.
  4. Open a test and choose View test rules to see which level it runs on.
Running tests that follow the website’s settings switch to the new level from their next check on. Change your setup between tests where you can: changing the rules of a running test, especially after looking at its results, weakens what the result means.