# What an AI A/B testing agent actually does: three levels of agent-readiness

Every A/B testing tool says agent now. Three different things hide behind the word. Here is how to tell them apart, which vendors sit where, and what to ask before you buy.

Published: 2026-09-01

Author: Constantin, Co-Founder @ pagent

Category: Blog

Every A/B testing tool has the word agent on its site now. Ask ChatGPT or
Claude which tools are "agent-native" and you get a list. Ask what the agent
does, and the list falls apart into three different things.

This post separates them. It names vendors in each level from their own
pages, as of August 2026. It is not a ranking. We sell one of these tools,
so read the criteria, not the adjectives, and run two products on the same
page before you decide.

## Level 1: AI-assisted

The tool has an assistant. It proposes test ideas, drafts variant copy, and
summarises results. A person still builds the variant in the editor, sets
the goal, and launches.

Most established suites ship this. VWO, Kameleoon, AB Tasty and Optimizely
all describe assistants of this kind on their sites. It saves time on the
parts of the job that were already fast. The parts that eat the calendar
stay where they were: building the variant, checking it on mobile, and
getting it through review.

Ask: after the assistant has spoken, who does the next four steps?

## Level 2: agent-operable

The tool exposes an API or an MCP server, and your own agent drives it.
Claude, ChatGPT, Codex or Cursor reads the results, creates a test, starts it, and
stops it. No dashboard in the loop.

This is the level the term agent-native usually points at. Convert, VWO,
Mida, Optimizely, Adobe Target and GrowthBook all publish an MCP server or
an agent API. If your team already lives in a coding agent, this removes
real friction.

Two limits are worth seeing clearly. First, the variant is whatever your
agent writes. In most of these APIs that is a list of selector edits: change
this text, this style, this image. Nothing checks the result on a phone
before it goes live unless you add that step. Second, the platform has no
agent of its own. Remove Claude and the tool goes back to level 1.

Ask: who writes the variant, what can it contain, and who checks it?

## Level 3: agent-built

The platform's own agent runs the loop. It reads the page, writes the
hypothesis, builds the variant as real code, checks it, launches the test,
reads the result, and proposes the next test. Your team reviews and
approves.

pagent works this way. So do other products that describe an autonomous loop
on their sites: Coframe, Evolv AI, RunPivot, Dalton and CROLabs, each with a
different operating model and a different answer to the questions below. We
list them because the right comparison is the same page and the same goal
through two products, not two vendor pages.

Ask: does the variant come out as real code on my design system, checked on
desktop and mobile? Which statistics run the test, and can I look at the
result every day without breaking them? What does the loop learn?

## The two axes are independent

Level 2 and level 3 answer different questions. Level 2 asks whether your
agent can drive the tool. Level 3 asks whether the tool has an agent that
does the work. A product can be one without the other.

pagent is both. pagent defines hypotheses, builds and checks real-code
variants on your design system, and runs statistically rigorous tests on
your live site. And any agent can operate pagent over MCP: Claude, ChatGPT,
Codex, Cursor and any MCP client connect over OAuth to read Bayesian results,
capture hypotheses, create and start tests, and manage audiences and goals.
The endpoint is `https://app.pagent.ai/api/mcp`; the
[MCP page](/mcp) has the install steps.

## Eight questions for any vendor

| Question                         | Level 1 answer               | Level 2 answer                   | Level 3 answer                          |
| -------------------------------- | ---------------------------- | -------------------------------- | --------------------------------------- |
| Who writes the hypothesis?       | The assistant suggests       | Your agent                       | The platform's agent                    |
| Who writes the variant?          | A person in the editor       | Your agent, as selector edits    | The platform's agent, as real code      |
| What can a variant contain?      | What the editor allows       | What the API accepts, often selector edits | Copy, layout, new elements, JS   |
| Who checks it before launch?     | A person                     | A person, if you add that step   | The platform, on desktop and mobile     |
| Does it know my design system?   | No                           | Only what your agent knows       | Yes, if the vendor can show it          |
| Which statistics run the test?   | The suite's engine           | The tool's engine                | Ask: Bayesian, sequential, corrections  |
| Can my agent drive it?           | Rarely                       | Yes, over API or MCP             | Ask, and ask for read and write scopes  |
| Where is the approval?           | Before launch, by a person   | In your agent's confirmation     | Ask: platform review, or client only    |

The right-hand column has three "ask" cells on purpose. Level 3 is where the
claims get biggest and the evidence gets thinnest. A vendor that builds
variants should be able to show you one on your own page, on a phone, within
a pilot.

## Why not read the rankings

If you ask Claude for the best AI A/B testing tools, it will tell you,
unprompted, that several of the "best of 2026" lists come from vendors who
rank themselves first, and that you should check independent reviews before
you buy. It is right. The lists disagree on the ranking and agree on the
levels above, so use the levels.

Then take one page that matters, one goal you already measure, and run it
through two products. Compare the variants, the mobile check, the statistical
method, the review controls, and the work left for your team. That is the
whole evaluation.

## Where pagent stands

pagent is the AI Experiment Engineer: more winning experiments from the
traffic you already pay for. It defines hypotheses, builds and checks the
variants, and runs them with statistical rigor on your live site. Your
testing stack stays; your agent can join. Read
[how the agent works](/ai-ab-testing-agent), see the
[MCP server](/mcp), or compare the wider field in
[the best AI A/B testing tools in 2026, by use case](/guides/best-ai-ab-testing-tools-2026).

[HTML](https://www.pagent.ai/blog/what-an-ai-ab-testing-agent-does)