> ## Documentation Index
> Fetch the complete documentation index at: https://www.pagent.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# How pagent decides a test

> The checks, rules and outcomes behind every test result

A test shows each visitor either the original page (**control**) or a **variation**. pagent counts how many visitors of each group complete the test's **primary goal**. From those counts, it decides whether the variation is better, worse or no different.

This page explains how that decision is made. The other pages in this section go deeper:

<CardGroup cols={2}>
  <Card title="Statistics settings" icon="sliders" href="/docs/statistics/settings">
    The five levels, every setting, its default, and when to change it.
  </Card>

  <Card title="Bayesian statistics" icon="percent" href="/docs/statistics/bayesian">
    Chance to beat control, priors and expected loss.
  </Card>

  <Card title="Frequentist statistics" icon="chart-line" href="/docs/statistics/frequentist">
    p-values, significance level and sequential correction.
  </Card>

  <Card title="Variations and goals" icon="layer-group" href="/docs/statistics/variations-and-goals">
    Tests with several variations, goals, revenue and traffic checks.
  </Card>

  <Card title="Choose your setup" icon="scale-balanced" href="/docs/statistics/choose-your-setup">
    Pick a level and match your company's standard.
  </Card>
</CardGroup>

## Method and level

Under **Settings → Tests → Statistics**, you make two choices:

1. **Method**: Bayesian or frequentist. Both use the same data and the same rules for runtime and data. They differ in the question they ask.
2. **Level**: from **Explore** (fastest, at most 1 in 7 false winners) to **Strict** (at most 1 in 100). The default is **Balanced**: at most 1 in 20 false winners among changes that do nothing.

See [Statistics levels](/docs/statistics/settings#statistics-levels) for what each level sets.

| | **Bayesian** (default) | **Frequentist** |
| - | - | - |
| Question | How likely is it that the variation is better? | If the variation did nothing, how surprising is this data? |
| Main number | Chance to beat control, for example 94 % | p-value, for example 0.012 |
| A winner is called when (Balanced) | The chance reaches 98.5 % | The p-value drops below the line for this check (significance level 10 %, two-sided) |
| A losing variation | Stops the test as **Stopped: variation was losing**, not as a result | Stops the test as a **Loss**, a result |
| Good for | Easy-to-read decisions | Teams that think in p-values and confidence levels |

Each website has a default method. A test takes it when it launches. You can switch a running or paused test to the other engine; see [Statistics settings](/docs/statistics/settings#statistics-engine).

## When pagent checks a test

pagent updates a test's numbers every hour. Updating the numbers never ends a test.

A test can only be ended at a **scheduled check**. By default there is one check a day, counted from the moment the test started. You can choose 1, 2, 4 or 24 checks a day with [Automatic checks per day](/docs/statistics/settings#automatic-checks-per-day). A check runs at the first hourly update after it is due.

The results page shows when the next check is due, under **Automatic decisions** and in the side panel under **Analysis → Next check**.

## The order of the checks

At every scheduled check, pagent goes through these rules in order. The first rule that applies decides what happens. The numbers are the Balanced defaults.

<Steps>
  <Step title="Enough data?">
    Control and the variation together need at least the [minimum total conversions](/docs/statistics/settings#minimum-total-conversions) (100). If not, the test keeps running.
  </Step>

  <Step title="Ran long enough?">
    The test must have run for the [minimum runtime](/docs/statistics/settings#minimum-runtime-days) (7 days). If not, the test keeps running.
  </Step>

  <Step title="Out of time?">
    Once the [maximum runtime](/docs/statistics/settings#maximum-runtime-days) (21 days) is reached, the test stops. It ends with a result if the final evidence passes the engine's bar, and as inconclusive otherwise. If steps 1 or 2 still fail at that point, it ends as inconclusive.
  </Step>

  <Step title="No meaningful difference?">
    If [practical equivalence](/docs/statistics/settings#practical-equivalence-rope) is on and the data shows the variation is almost certainly within ±7.5 % of control, the test stops as **inconclusive**. Further testing would not find a difference worth acting on.
  </Step>

  <Step title="Clear winner?">
    If the evidence passes the bar, the variation **wins** and the test stops.
  </Step>

  <Step title="Clearly losing?">
    Under the one-sided [decision policy](/docs/statistics/settings#decision-policy) (Bayesian default), the test stops when the variation's chance to beat control falls to 5 % or less, recorded as **Stopped: variation was losing**. Under the two-sided policy (frequentist default), a clear loser is a **Loss**.
  </Step>

  <Step title="Small enough risk? (Bayesian, Careful and Strict)">
    If the [expected loss check](/docs/statistics/settings#enable-expected-loss-threshold) is on, a Bayesian result also needs a small expected loss before the test stops.
  </Step>
</Steps>

If no rule ends the test, it keeps running until the next check.

## Possible results

| Result | Meaning | Counts as a learning |
| - | - | - |
| **Win** | The variation did better than the original on the primary goal. | Yes |
| **Loss** | The variation did worse than the original. Two-sided tests only. | Yes |
| **Stopped: variation was losing** | A one-sided test stopped because the variation was clearly losing. It protects your conversions, but it is not a statistical result. | No |
| **Inconclusive** | No clear difference either way. This includes tests stopped for practical equivalence, and tests that reached their maximum runtime without clear evidence. | Yes |

The test list has tabs for **Lost**, **Stopped: losing**, **Inconclusive** and **Failed** (tests that errored or were aborted).

## Who decides the result

Every finished test shows who decided its result.

* **Decided by pagent.** pagent ended the test at a scheduled check, at the maximum runtime, or because no visitors arrived (see [Maximum idle days](/docs/statistics/settings#maximum-idle-days)).
* **Decided by you.** You stopped the test yourself, or changed the result afterwards.

**Stopping a test yourself.** Admins can open a running test and choose **Stop test**. The dialog shows **pagent's current reading** and asks for **Your result**: Win, Inconclusive or Loss. If you don't pick one, it is recorded as inconclusive. pagent does not decide a result when you stop a test, because it only decides at its scheduled checks.

**Changing a result.** On a finished test, admins can choose **Change result…** in the test's menu. Your result then counts instead of pagent's, also for a test that was stopped because the variation was losing. Reports and learnings show that you decided it. **Use pagent's verdict** switches back.

**Shipping is a separate decision.** A result never ships anything by itself. Use **Mark as shipped…** when you have built a variation into your site.

## How runtime is counted

* Runtime starts when the test launches and is counted in days, including parts of a day.
* **Paused time counts.** Pausing a test does not stop its clock.
* If you resume a test after its maximum runtime has passed, it ends right away.

## Tests with more than one variation

Each variation is compared with control on its own. pagent makes the bar for a winner stricter for each comparison, so that more variations do not mean more false winners. When the best variation's comparison ends the test, the whole test stops for all variations. See [Variations and goals](/docs/statistics/variations-and-goals#tests-with-several-variations).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.