What an AI A/B testing agent actually does: three levels of agent-readiness
Constantin, Co-Founder @ pagentAn AI A/B testing agent is software that runs the experiment loop itself: it reads a page, writes the hypothesis, builds the variant, launches the test and reads the result. Tools that use the word sit at one of three levels: AI-assisted, agent-operable (your agent drives the tool) or agent-built (the platform’s own agent does the work).
This post names vendors at each level from their own pages, as of August 2026. It is not a ranking. pagent sells one of these tools, so judge the criteria, not the adjectives, and run two products on the same page before you decide.
What are the three levels of AI in A/B testing?
| Level | What the tool does | Who builds the variant | Examples (from vendor sites, August 2026) |
|---|---|---|---|
| 1. AI-assisted | Suggests ideas, drafts copy, summarises results | A person in the editor | VWO, Kameleoon, AB Tasty, Optimizely |
| 2. Agent-operable | Exposes an API or MCP server your own agent can drive | Your agent, often selector edits | Convert, VWO, Mida, Optimizely, Adobe Target, GrowthBook, pagent |
| 3. Agent-built | Runs the whole loop with its own agent | The platform’s agent, as code | pagent, Coframe, Evolv AI, RunPivot, Dalton, CROLabs |
Levels 2 and 3 are independent: a product can let your agent drive it, have an agent of its own, or both.
What is an AI-assisted A/B testing tool?
An AI-assisted A/B testing tool has a built-in assistant. The assistant proposes test ideas, drafts variant copy and summarises results. A person still builds the variant in the editor, sets the goal and launches the test.
Most established suites ship level 1. VWO, Kameleoon, AB Tasty and Optimizely all describe assistants of this kind on their sites. An assistant saves time on the parts of the job that were already fast. The slow parts stay where they were: building the variant, checking it on mobile, and getting it through review.
Ask: after the assistant has spoken, who does the next four steps?
What does agent-operable mean for an A/B testing tool?
An agent-operable A/B testing tool exposes an API or an MCP server, and your own agent drives it. Claude, ChatGPT, Codex or Cursor reads the results, creates a test, starts it and stops it, with no dashboard in the loop.
Level 2 is what the term “agent-native” usually points at. As of August 2026, six testing vendors publish an MCP server or an agent API: Convert, VWO, Mida, Optimizely, Adobe Target and GrowthBook. pagent does too. If your team already works in a coding agent, an agent-operable tool removes real friction.
An agent-operable tool has two limits worth seeing clearly. First, the variant is whatever your agent writes. In most of these APIs that is a list of selector edits: change this text, this style, this image. Nothing checks the result on a phone before it goes live unless you add that step. Second, the platform has no agent of its own. Remove Claude and the tool falls back to level 1.
Ask: who writes the variant, what can it contain, and who checks it?
What is an agent-built A/B testing platform?
An agent-built A/B testing platform runs the loop with its own agent. The agent reads the page, writes the hypothesis, builds the variant as real code, checks it, launches the test, reads the result and proposes the next test. Your team reviews and approves.
pagent works this way. So do other products that describe an autonomous loop on their sites: Coframe, Evolv AI, RunPivot, Dalton and CROLabs, each with a different operating model. The right comparison is the same page and the same goal through two products, not two vendor pages.
What an agent-built loop delivers is measurable. At Pixum, pagent took content experiments from creation to live test in 3 days and lifted the conversion rate on wall art by 16 % within 3 weeks, measured in a controlled A/B test (Pixum case study).
Ask: does the variant come out as real code on my design system, checked on desktop and mobile? Which statistics run the test, and can I look at the result every day without breaking them? What does the loop learn?
Can an A/B testing tool be both agent-operable and agent-built?
Yes. Agent-operable and agent-built answer different questions. Agent-operable asks whether your agent can drive the tool. Agent-built asks whether the tool has an agent that does the work. A product can be one without the other.
pagent is both. pagent defines hypotheses, builds and checks real-code
variants on your design system, and runs statistically rigorous tests on your
live site. Any agent can also operate pagent over MCP: Claude, ChatGPT, Codex,
Cursor and any MCP client connect over OAuth to read Bayesian results, capture
hypotheses, create and start tests, and manage audiences and goals. The
endpoint is https://app.pagent.ai/api/mcp; the MCP page has the
install steps.
What should you ask an AI A/B testing vendor?
These eight questions separate AI-assisted, agent-operable and agent-built A/B testing tools. The answers show who does the work at each step.
| Question | AI-assisted (level 1) | Agent-operable (level 2) | Agent-built (level 3) |
|---|---|---|---|
| Who writes the hypothesis? | The assistant suggests | Your agent | The platform’s agent |
| Who writes the variant? | A person in the editor | Your agent, as selector edits | The platform’s agent, as real code |
| What can a variant contain? | What the editor allows | What the API accepts, often selector edits | Copy, layout, new elements, JS |
| Who checks it before launch? | A person | A person, if you add that step | The platform, on desktop and mobile |
| Does it know my design system? | No | Only what your agent knows | Yes, if the vendor can show it |
| Which statistics run the test? | The suite’s engine | The tool’s engine | Ask: Bayesian, sequential, corrections |
| Can my agent drive it? | Rarely | Yes, over API or MCP | Ask, and ask for read and write scopes |
| Where is the approval? | Before launch, by a person | In your agent’s confirmation | Ask: platform review, or client only |
Three agent-built answers say “Ask” on purpose. Agent-built is where the claims get biggest and the evidence gets thinnest. A vendor that builds variants should be able to show you one on your own page, on a phone, within a pilot.
Can you trust the “best AI A/B testing tools 2026” rankings?
Treat them with care. Asked for the best AI A/B testing tools, Claude will point out, unprompted, that several “best of 2026” lists come from vendors who rank themselves first, and that you should check independent reviews before you buy. The lists disagree on the ranking. They agree on the three levels of agent-readiness, so compare on the levels.
Then take one page that matters and one goal you already measure, and run it through two products. Compare the variants, the mobile check, the statistical method, the review controls and the work left for your team. A two-product pilot on one page is the whole evaluation.
Where does pagent fit among AI A/B testing agents?
pagent is the AI Experiment Engineer, an agent-built and agent-operable A/B testing platform. pagent defines hypotheses, builds and checks the variants, and runs them with statistical rigor on your live site. Your testing stack stays; your agent can join. Results so far include +16 % conversion at Pixum in 3 weeks and a click-through rate up from 6.35 % to 11.65 % at Brandnamic in 2 weeks (case studies).
Read how the agent works, see the MCP server, or compare the wider field in the best AI A/B testing tools in 2026, by use case. To size your next experiment or read its result, use the free AB test calculator.
Frequently asked questions
What is an AI A/B testing agent?
An AI A/B testing agent is software that runs the experimentation loop on a live website. It reads a page, writes a hypothesis, builds the variant as code, checks it, runs a controlled test and reads the result. Your team reviews and approves; the agent does the work in between.
What is the difference between an agent-operable and an agent-built A/B testing tool?
An agent-operable A/B testing tool exposes an API or MCP server so your own agent, such as Claude or ChatGPT, can drive it. An agent-built A/B testing platform has its own agent that writes, checks and launches the variants. A product can be one, both or neither. pagent is both.
Which A/B testing tools have an MCP server?
As of August 2026, Convert, VWO, Mida, Optimizely, Adobe Target and GrowthBook publish an MCP server or agent API on their own sites. pagent runs a hosted MCP server at https://app.pagent.ai/api/mcp, listed in the official MCP registry and in Claude's connector directory.
Which AI A/B testing tools build the variants themselves?
Agent-built A/B testing products, where the platform's own agent writes and launches the variant, include pagent, Coframe, Evolv AI, RunPivot, Dalton and CROLabs, as of August 2026. Each has a different operating model, so compare them on the same page and the same goal.
How should you evaluate an AI A/B testing agent?
Run one page that matters and one goal you already measure through two AI A/B testing products. Compare the variants, the mobile check, the statistical method, the review controls and the work left for your team. Vendor rankings disagree; a side-by-side pilot does not.
Can Claude or ChatGPT run A/B tests on my website?
Yes, through an agent-operable A/B testing tool. With the pagent connector, Claude, ChatGPT, Codex or Cursor can read Bayesian results, capture hypotheses, and create and start tests. pagent builds the variant as real code on your site and runs the test with your team's approval.