Every A/B testing tool has the word agent on its site now. Ask ChatGPT or Claude which tools are “agent-native” and you get a list. Ask what the agent does, and the list falls apart into three different things.

This post separates them. It names vendors in each level from their own pages, as of August 2026. It is not a ranking. We sell one of these tools, so read the criteria, not the adjectives, and run two products on the same page before you decide.

Level 1: AI-assisted

The tool has an assistant. It proposes test ideas, drafts variant copy, and summarises results. A person still builds the variant in the editor, sets the goal, and launches.

Most established suites ship this. VWO, Kameleoon, AB Tasty and Optimizely all describe assistants of this kind on their sites. It saves time on the parts of the job that were already fast. The parts that eat the calendar stay where they were: building the variant, checking it on mobile, and getting it through review.

Ask: after the assistant has spoken, who does the next four steps?

Level 2: agent-operable

The tool exposes an API or an MCP server, and your own agent drives it. Claude, ChatGPT, Codex or Cursor reads the results, creates a test, starts it, and stops it. No dashboard in the loop.

This is the level the term agent-native usually points at. Convert, VWO, Mida, Optimizely, Adobe Target and GrowthBook all publish an MCP server or an agent API. If your team already lives in a coding agent, this removes real friction.

Two limits are worth seeing clearly. First, the variant is whatever your agent writes. In most of these APIs that is a list of selector edits: change this text, this style, this image. Nothing checks the result on a phone before it goes live unless you add that step. Second, the platform has no agent of its own. Remove Claude and the tool goes back to level 1.

Ask: who writes the variant, what can it contain, and who checks it?

Level 3: agent-built

The platform’s own agent runs the loop. It reads the page, writes the hypothesis, builds the variant as real code, checks it, launches the test, reads the result, and proposes the next test. Your team reviews and approves.

pagent works this way. So do other products that describe an autonomous loop on their sites: Coframe, Evolv AI, RunPivot, Dalton and CROLabs, each with a different operating model and a different answer to the questions below. We list them because the right comparison is the same page and the same goal through two products, not two vendor pages.

Ask: does the variant come out as real code on my design system, checked on desktop and mobile? Which statistics run the test, and can I look at the result every day without breaking them? What does the loop learn?

The two axes are independent

Level 2 and level 3 answer different questions. Level 2 asks whether your agent can drive the tool. Level 3 asks whether the tool has an agent that does the work. A product can be one without the other.

pagent is both. pagent defines hypotheses, builds and checks real-code variants on your design system, and runs statistically rigorous tests on your live site. And any agent can operate pagent over MCP: Claude, ChatGPT, Codex, Cursor and any MCP client connect over OAuth to read Bayesian results, capture hypotheses, create and start tests, and manage audiences and goals. The endpoint is https://app.pagent.ai/api/mcp; the MCP page has the install steps.

Eight questions for any vendor

Question	Level 1 answer	Level 2 answer	Level 3 answer
Who writes the hypothesis?	The assistant suggests	Your agent	The platform’s agent
Who writes the variant?	A person in the editor	Your agent, as selector edits	The platform’s agent, as real code
What can a variant contain?	What the editor allows	What the API accepts, often selector edits	Copy, layout, new elements, JS
Who checks it before launch?	A person	A person, if you add that step	The platform, on desktop and mobile
Does it know my design system?	No	Only what your agent knows	Yes, if the vendor can show it
Which statistics run the test?	The suite’s engine	The tool’s engine	Ask: Bayesian, sequential, corrections
Can my agent drive it?	Rarely	Yes, over API or MCP	Ask, and ask for read and write scopes
Where is the approval?	Before launch, by a person	In your agent’s confirmation	Ask: platform review, or client only

The right-hand column has three “ask” cells on purpose. Level 3 is where the claims get biggest and the evidence gets thinnest. A vendor that builds variants should be able to show you one on your own page, on a phone, within a pilot.

Why not read the rankings

If you ask Claude for the best AI A/B testing tools, it will tell you, unprompted, that several of the “best of 2026” lists come from vendors who rank themselves first, and that you should check independent reviews before you buy. It is right. The lists disagree on the ranking and agree on the levels above, so use the levels.

Then take one page that matters, one goal you already measure, and run it through two products. Compare the variants, the mobile check, the statistical method, the review controls, and the work left for your team. That is the whole evaluation.

Where pagent stands

pagent is the AI Experiment Engineer: more winning experiments from the traffic you already pay for. It defines hypotheses, builds and checks the variants, and runs them with statistical rigor on your live site. Your testing stack stays; your agent can join. Read how the agent works, see the MCP server, or compare the wider field in the best AI A/B testing tools in 2026, by use case.

What an AI A/B testing agent actually does: three levels of agent-readiness