Pip: the agent harness we built by taking things away
Christopher, Founder @ pagentPip is the agent harness that builds A/B tests in pagent. It turns a hypothesis into a working change on a customer’s live website, checks it in a real browser, and fixes it from review feedback. Pip has a 30-word role prompt and a handful of tools. It took us three years and four architectures to get there. Along the way, pagent’s agents built more than 2,000 tests, with almost 8,000 variants and about 25,000 changes, on 672 websites.
Almost every improvement in this story came from taking something away: rules the model had to work around, hand-offs between agents, pages it had to find again, and time spent between turns. What we did add was capability, mostly as options on Pip’s tools. In a controlled test, each of nine added instructions broke something elsewhere.
You may have met Pip before. Pip is the small character that walks new visitors through their first test in pagent. That first test used to be built by a team of specialised agents. Now Pip builds it, and every test after it.
What does an agent harness have to do for pagent?
Everybody defines a harness a little differently. Earendil describes it as storage plus the machinery to run conversations with language models, the tools those models call, and the environments the tools run in. We agree, and we would add that a harness also contains every opinion you have about how the model should work. In our case, those opinions caused most of the problems.
pagent’s job, running the A/B testing loop from hypothesis to result, is unusual for an agent. We do not write code in a repository we control. We change somebody else’s live website, while it runs, for a share of its visitors. Such a website shows cookie dialogs and bot walls. Its framework re-renders the page and wipes our change, the layout is different on phones, and a product editor can have state that only exists after six clicks.
A change that pagent ships has to apply without flicker, survive the site’s own scripts, work on phones and desktops, and be reversible. And before any visitor sees it, a person approves it.
Pip turns the idea into a change: text, layout, images or new elements, and where on the site it runs.
Pip does all of this in one conversation per test. It builds a change, opens the page in a browser, looks at it as a phone and as a desktop, and submits it for review. When a reviewer asks for something, the feedback arrives in the same conversation, and Pip fixes the change with everything it learned while building it.
Three years, four architectures
Each architecture made sense for the models of its time. We started with an agent, took the decisions away from it when it was not good enough, gave them back one step at a time, and ended with a single agent again.
One agent, one conversation per test, a handful of tools. It builds, checks, takes review feedback and fixes, without hand-offs.
Show the numbers
| Era | When | What it was |
|---|---|---|
| An agent | Sep 2023 – Jul 2024 | One agent reads the page and proposes changes. A manager agent is gone after two months; the agent is rewritten twice. The models are not reliable enough to plan, act and check on their own. |
| A pipeline | Aug 2024 – Aug 2025 | Fixed steps: plan, write, critique, execute. Each step is one model call with structured output. Predictable and easy to debug, but no step can look at the result and change its mind. |
| Specialised agents | Sep 2025 – Oct 2026 | Models get good at calling tools, and each step becomes an agent: one builds the variation, one reviews it visually, one reviews it against the hypothesis, one fixes review findings. About a dozen agents in one package. |
| A single agent: Pip | Oct 2026 – | One agent, one conversation per test, a handful of tools. It builds, checks, takes review feedback and fixes, without hand-offs. |
Figure: how pagent built its tests. An agent from September 2023 to July 2024. A pipeline of fixed steps from August 2024 to August 2025. Specialised agents from September 2025 to October 2026. A single agent, Pip, since October 2026, after two harness experiments in September and October 2026.
2023: an agent, too early
pagent’s first commit for generating changes, in September 2023, was an agent. It read a page’s HTML and proposed changes to it. A manager agent handed work to sub-agents, and within two months we removed the manager. The agent was rewritten twice in its first half year. Much of the work went into keeping it inside its limits: trimming HTML so the page fit the context window, raising timeouts from 30 to 60 seconds, running calls in parallel so a test did not take minutes.
The models of that time could write a convincing change. They could not reliably plan it, carry it out and check it on their own.
2024: a pipeline, because the agents were not good enough
So in August 2024 we took the decisions away from the model. The agent became a pipeline of fixed steps: plan the change, write it, critique it, execute it. Each step was one model call with a template prompt and structured output. By December every generation ran on the second version of that pipeline, and through 2025 it grew new steps for planning sections, adjusting copy and applying a customer’s guidelines.
A pipeline is what Anthropic calls a workflow rather than an agent, and it is predictable. You can trace it, test each step and debug a bad result. But no step can look at the result and change its mind. A step can only work with what the step before handed it, and a page that did not match the plan had nowhere to go.
2025–2026: specialised agents
By the second half of 2025 the models had become good at calling tools, and we gave the decisions back, step by step. In September 2025 we retired the pipeline. Each step became an agent with its own tools, prompt and guidelines: one agent built the variation, one reviewed it visually in a browser, one reviewed it against the hypothesis, one fixed review findings. By March 2026 there were about a dozen agents in one package.
Each agent made sense on its own, but every hand-off between them lost context. The reviewer did not know what the builder had tried, and each agent kept separate rules and a separate budget. It worked often enough to run tests for customers, and it failed in ways we kept fixing one ticket at a time.
Show the numbers
| Failure | Share |
|---|---|
| Wrong element or content changed | 18% |
| Build failed or produced no change | 15% |
| Review loop stuck or feedback not applied | 13% |
| Flicker, or the site re-rendered the change away | 13% |
| Brittle selectors | 9% |
| Timeouts and stuck builds | 9% |
| Cookie dialogs and bot walls | 9% |
| Copy too long or off-tone | 7% |
| Missed on phones only | 7% |
Figure: generation-failure tickets by kind. Wrong element or content changed: 18%. Build failed or produced no change: 15%. Review loop stuck or feedback not applied: 13%. Flicker or re-render: 13%. Brittle selectors, timeouts, and cookie dialogs or bot walls: 9% each. Copy too long or off-tone, and misses on phones only: 7% each.
The cookie banner. In a demo for a sports club, almost half of the test builds ended in an error at the same step, with no readable reason. They had not crashed. The review agent had chosen to stop. A cookie banner covered the hero in every screenshot the agent took, and the agent had used up its three visual reviews trying to see past the banner. Over the next two weeks we merged about a dozen fixes to the browser review step. The last one was titled “End test builds whose browser review loops”.
The rule that froze a page. For a telco’s pricing cards, we benchmarked the agent on one hard variation: show monthly and yearly prices behind a toggle. A rule written as advice (“guard your observer”) was ignored, and the generated script watched the container it wrote into. The first click on the toggle froze the page in an endless render loop. Once we wrote it as a mandatory rule, the bug went away, but the variant looked less polished than before. It took a day of comparisons to see why: the old, nicer-looking variant had shown static prices that were simply wrong. The agent’s own write-up put it well: “The compactness was bought with the incorrectness.”
The change that never applied. In October, a conversion manager at a photo-products retailer tried to build a test in pagent. The requested changes were not applied. The manager tried several more times, rewording the request each time, and then stopped. The build was stuck in “Needs input”, because the automatic browser check had found a problem it could not fix.
Each of these failures had a fix, but every fix added a rule, a step or a budget, and the fixes interacted. From our planning notes in August: “we organically developed a harness in our current product which is not ideal.”
Detour: a harness that improves itself
In September 2026 we started again from a clean slate, in a separate repository. The idea was to let the harness improve itself, which is now often called autoresearch after Andrej Karpathy’s project of that name: agents propose a change to the harness, run it against evaluation cases, and keep it if the score goes up.
This time we read first: papers on self-improving agents, books on agent design and other teams’ harnesses. We wrote a wiki of sixteen chapters before the first line of code, opening with “twenty laws”. Some of them are still the best summary of what we believe:
Less structure, more intelligence. Hard-code only boundaries that must always hold. Give the decision space back to the model.
Verification the model can skip is not verification.
Ten focused tools beat fifty overlapping ones.
Then we built it, mostly with coding agents writing the code, and it grew faster than we could evaluate it.
In eighteen days the codebase grew seventy-fold, to a quarter of a million lines in forty packages. It had a learning loop that turned review feedback into lessons, a knowledge base with consolidation and curation, a synthetic audience to rank variants, generated sub-agents, a kernel that agents could extend, and a way to ship winning variants into a customer’s CMS. A catalogue we wrote at the end lists 370 ideas. Two in three of them were built.
It also broke its own laws. The laws preferred a few focused tools; the builder ended up with about thirty. “Less structure, more intelligence” sat next to a builder prompt that nearly tripled.
And much of it never met reality. The README said so: “All experiment outcomes are simulated, and will be for a considerable time.” The knowledge base recorded that not one of its entries had ever been read by an agent. The harness ran a self-audit, and on the day all but one of its roadmap items were marked done, the audit said: “Health: Degraded”.
We also could not tell which of the 250 built ideas helped, so the loop had nothing to climb.
Hill climbing on something small
So inside that same repository, we built something small to climb on. It had one conversation and one browser page, and no knowledge base or sub-agents. It ran on six real cases where a customer had approved a variant, each with a rubric of checks. Coding agents ran a hill-climbing loop: look at the failures, propose a change, run the cases, keep the change only if no check gets worse. The small harness went through 32 versions in three days.
The obvious lever was the prompt. Nine times, the loop looked at the mistakes and wrote an instruction against them, each between 52 and 169 words, each aimed at a failure it had seen.
Show the numbers
| Addition | Words added | Unchanged prompt | With addition | What broke |
|---|---|---|---|---|
| #1 | 52 | 5 / 12 | 5.5 / 12 | Mobile layout |
| #1 | 52 | 6.5 / 12 | 6 / 12 | Mobile layout |
| #2 | 90 | 9 / 12 | 10 / 12 | Keyboard use of the control |
| #3 | 62 | 8.5 / 12 | 10.5 / 12 | Mobile layout, keyboard use |
| #3 | 62 | 9 / 12 | 10.5 / 12 | Mobile layout, keyboard use |
| #4 | 64 | 9.5 / 12 | 11 / 12 | Delivery details |
| #5 | 68 | 9 / 12 | 10 / 12 | Keyboard use of the control |
| #6 | 118 | 9.5 / 12 | 11 / 12 | Mobile layout |
| #7 | 80 | 9 / 12 | 10 / 12 | Delivery details |
| #7 | 80 | 9 / 12 | 9.5 / 12 | Delivery details |
| #8 | 169 | 8.5 / 12 | 11.5 / 12 | Product order, other case |
| #8 | 169 | 10.5 / 12 | 12 / 12 | Product order, other case |
| #8 | 169 | 9 / 12 | 11.5 / 12 | Product order, other case |
| #8 | 169 | 7 / 8 | 8 / 8 | Product order, other case |
| #8 | 169 | 7 / 8 | 7 / 8 | Product order, other case |
| #8 | 169 | 7 / 8 | 5 / 8 | Product order, other case |
| #9 | 99 | 9 / 12 | 10.5 / 12 | Mobile labels cut off |
Figure: nine prompt additions tested against the unchanged prompt. Most raised the rubric score, for example from 71% to 96% of the points on one case, and every one broke a check somewhere else: mobile layout, keyboard use of a control, delivery details, or product order on a different case. None was accepted.
Most of them raised the total score, sometimes by a lot, and every one broke something else. An instruction about layout made the agent rebuild a control and lose its keyboard behaviour. An instruction about native controls made it drop delivery details. The best one brought three runs of one case close to perfect and then reordered products on another case. The prompt we kept had nothing added to it.
The unchanged prompt scored anywhere between 71% and 88% on the same case, so most steps were smaller than the noise. Each instruction was written to fix the case in front of it, and it overfit to that case. And the hill itself kept moving: six of the 32 versions were fixes to our own grading, not to the agent.
In a side test during the same three days, with a small, fast model driving the browser, the model found the right button every time but finished cleanly in only 60% of the runs. With a completion check written in code, it finished in 95%. Our notes on the test: “The main model weakness was stopping, not finding the button.”
The last commit in that repository is the catalogue of all 370 ideas, with what each was for and whether it worked. We kept the lessons and left the code. The next harness would not improve itself, and every idea would have to prove itself against the current version.
A single agent: start with almost nothing
In early October we started again, with a different rule for ourselves: one agent, the least harness that can do the job, and one idea at a time, only with evidence.
Why we moved from the Vercel AI SDK to pi-durable
Until then, our agents ran on the Vercel AI SDK. It served us well for single model calls and for the pipeline. For a long-running agent it was the wrong shape: the SDK owned the loop between the model and its tools. Our refactoring notes from September read: “the loop could not be observed or changed.” So the self-improving harness got a loop of our own, modelled on Pi, the minimal coding agent from Earendil. It worked, but a durable agent loop is a product of its own, and maintaining one was not the job we wanted to be doing.
For Pip, we stopped maintaining our own loop. Pip is built on pi-durable, Earendil’s framework for long-running agents, for these reasons:
- Minimal on purpose. Pi only adopts a feature after it has proven itself and weighs it against the complexity it adds, as Earendil describes in its Pi 1.0 announcement. That is exactly the rule we wanted for our own harness.
- Extensions you can add and remove. An extension bundles tools, prompt sections and hooks under a name. Adding an idea means adding an extension; removing it means deleting a folder. An extension can replace or wrap an existing tool without touching the core. That way the code itself keeps every idea removable.
- Durable by default. A pagent test lives for days: it builds, waits for a reviewer, gets feedback, and builds again, across many short serverless steps. pi-durable checkpoints every step, so a conversation survives a crash and continues where it stopped.
- The transcript is the truth. pi-durable records every change to the prompt where it happened, so a restart sees exactly what the model saw, and a stable cache key keeps the prompt cache warm.
- Small enough to read. The whole framework is about 15,000 lines, which our own coding agents can read when they build on it.
And a small bias we will admit to: Pi comes from Austria, and so does the author of this post.
With pi-durable running the loop, we only had to write the part that is specific to pagent: the tools, the browser and the review loop.
Two sentences and a browser
On day one, Pip had a role prompt of two sentences:
You build one variation of an A/B test on a live website. Turn the instruction into a change, check it in the browser, and refine it when humans give feedback.
It could open the page in a browser, submit a change, and read the files attached to the test. The rules file for contributors starts with a heading in capitals:
DO NOT LOBOTOMIZE THE MODEL. For any change: do not give the model rules it has to work around, and do not make it dumber. Add capability, context or evidence.
The rest of the rules make that enforceable:
- One idea per pull request, stated as a hypothesis: “Adding X makes the agent better at Y.”
- It must improve test quality, time or cost, and the pull request shows how.
- Ship it as an extension. Deleting the extension must restore the previous baseline.
- Prefer an option on an existing tool over a new tool.
- Every merged extension is active for every test. A weak idea makes every test worse; there are no switches.
- Bring evidence. Run the same instruction on the current version and on the branch, and compare quality, time, tokens and cache rate.
- No autonomous self-improvement. People choose which idea to test next.
The instructions we compare on come from production: real hypotheses that pagent had already tested for customers, saved as templates so that every comparison starts from the same instruction.
After three weeks of pull requests, one idea each, this is the prompt the model reads before it starts.
Instructions: 30 words until version 0.8, then 56. Extensions: one, then six.
Show the numbers
| Version | Instructions | Tool docs | Extensions | What it added |
|---|---|---|---|---|
| 0.1 | 30 | 261 | 1 | Start: browser, submit, attachments |
| 0.2 | 30 | 457 | 1 | Triggers: where a change runs |
| 0.3 | 30 | 536 | 1 | Asking a person |
| 0.4 | 30 | 676 | 2 | Images |
| 0.6 | 30 | 831 | 2 | Changes in steps |
| 0.8 | 56 | 880 | 3 | Phones and desktops |
| 0.8.1 | 56 | 900 | 3 | Triggers on any host |
| 0.9 | 56 | 900 | 4 | Clicks that say why |
| 0.12 | 56 | 926 | 6 | Browser follows a journey |
| 0.14 | 56 | 1092 | 6 | Site images, compositions, batches |
| 0.15 | 56 | 1207 | 6 | Native crops and sizes |
Figure: words the model reads before it starts, per Pip version. Instructions stayed between 30 and 56 words. The documentation of tools and their options grew from 261 to 1,207 words. Pip went from one extension to six.
The prompt grew 4.3 times, almost all of it in tool documentation. Of the 1,263 words, 56 are instructions: the 30-word role, and one sentence of context added in version 0.8. Everything else describes what the tools can do. New capabilities almost always became an option on a tool Pip already had, rather than a new tool.
The one line of context is a good example of the difference between context and a rule. Before version 0.8, Pip’s browser only opened desktop pages. On one test it reported that the responsive sizes were in place without ever having looked at the page on a phone. We did not add “always check mobile”. We gave the browser a device option and added one sentence: “Visitors reach the test’s site on phones and on desktops, and the change runs for both.” On the same test, Pip opened the page as a phone and got the sizes right, at the same cost.
Asking a person works the same way. Pip can ask a question, but the option’s description explains why that is the last resort: waiting on a person stalls the test until they answer, while a sensible choice gets corrected in review anyway. Faced with an instruction to use a slogan nobody had written yet, Pip asked for the slogan instead of inventing one.
And once, we took a guardrail out. Pip’s browser refused to run a step of a change on a different host than the test’s page. On an airline’s site, where the booking flow lives on a shop subdomain, that rule made Pip work around the harness. The pull request removing it says: “That’s a guardrail the agent had to work around, so this removes it.”
What worked: stop making the agent rediscover things
We found the best improvements by reading transcripts.
The browser forgot everything. Reading transcripts from a day of real tests, we noticed that in three out of four transcripts, Pip had at some point looked at an empty page. A long test runs as a series of short, durable steps, and each step reconnected to a fresh browser session that had lost the page, the cookies and the phone view. The agent spent turns finding out where it was. We let the browser session outlive the step.
Clicks failed slowly. Many of Pip’s clicks that day waited the full 15 seconds before failing, because the element did not exist or something covered it. Now a click checks first, and when it is certain the click cannot work, it says why at once: what covers the element, or that nothing matches.
The browser did not follow a journey. On a multi-page test, the page changed under the agent while it was reading it, and its reads failed with errors. Now the browser behaves like a visitor’s tab across the journey and reports where the page went.
Show the numbers
| Change | Measure | Before | After | Difference |
|---|---|---|---|---|
| Browser survives between steps | Time | 100 | 87 | −13% |
| Browser survives between steps | Cost | 100 | 88 | −12% |
| Browser survives between steps | Prompt tokens | 100 | 67 | −33% |
| Browser survives between steps | Tool calls | 100 | 75 | −25% |
| Browser survives between steps | Tool errors | 100 | 45 | −55% |
| Clicks that say why | Clicks that waited 15 s | 100 | 38 | −62% |
| Clicks that say why | Time in failed clicks | 100 | 58 | −42% |
| Clicks that say why | Time | 100 | 95 | −5% |
| Clicks that say why | Cost | 100 | 121 | +21% |
| Browser follows a journey | Time | 100 | 81 | −19% |
| Browser follows a journey | Cost | 100 | 82 | −18% |
| Browser follows a journey | Browser errors | 100 | 29 | −71% |
| Browser follows a journey | Page changed mid-call | 100 | 0 | −100% |
Figure: keeping the browser alive between steps cut time by 13%, prompt tokens by 33% and tool errors by 55%. Clicks that say why cut clicks waiting the full 15 seconds by 63% and time in failed clicks by 42%, while costing 21% more on that run. Following the journey cut time by 19% and browser errors by 71%.
We merged the click change for the time it saved on failed clicks, although its run cost 21% more, and said so in the pull request: “I’m not claiming a token improvement.”
None of the three changes tells the model what to do. They make the browser it works in behave more predictably.
What did not work
Most ideas did not survive. In one week in late October we ran about twenty experiments against the same templates. Fewer than a third shipped.
Show the numbers
| Idea | Cost | Time | Outcome |
|---|---|---|---|
| Higher reasoning effort | +150% | — | Dropped |
| A proxy that compresses context | +114% | +47% | Dropped |
| Clicks that say why | +21% | −5% | Shipped |
| Hide old tool results | +15% | — | Dropped |
| Several browser actions per call | +10% | +17% | Dropped |
| A ledger of review items in the prompt | +10% | — | Dropped |
| A virtual filesystem for the change | +9% | — | Dropped |
| Filter large results by intent | +5% | — | Dropped |
| Mechanical checks after every submit | −8% | +14% | Dropped |
| Browser survives between steps | −12% | −13% | Shipped |
| Browser follows a journey | −18% | −19% | Shipped |
Figure: change in cost against the unchanged harness. Dropped: higher reasoning effort (+150%), a context-compressing proxy (+114%, time +47%), hiding old tool results (+15%), several browser actions per call (+10%, time +17%), a review ledger (+10%), a virtual filesystem (+9%), filtering large results (+5%), mechanical checks after every submit (−8%, time +14%). Shipped: clicks that say why (+21%, time −5%), a browser that survives (−12%, time −13%), a browser that follows a journey (−18%, time −19%). Identical runs vary by about ±9% in cost.
Why the dropped ideas failed:
- Several browser actions per call: fewer turns should mean less cost. In practice, the agent “spent the saved turns on more actions instead of finishing sooner”.
- A virtual filesystem for the change: Pip’s submissions got 81% shorter, but it needed 22% more turns, and each extra turn re-reads about 29,000 tokens of conversation.
- Hiding old tool results: a replay predicted 10–14% savings. In real runs it cost 15% more, because changing earlier context breaks the prompt cache.
- A context-compressing proxy: cost more than doubled. “Its tools were never used.”
- Reporting image weight: Pip was told “610 KB, shown 150 px wide” eight times in one run and ignored it every time. What worked later was letting Pip crop and resize images itself.
- Checks written by the agent: Pip wrote replayable checks for its own change. They failed on correct changes, and runs took 30% longer.
- More reasoning: the same results at about 2.5 times the cost.
Profiling the whole run
Durability has a price. Every turn writes a checkpoint, and a long test runs as many short steps that each have to pick up where the last one stopped. Later in October we timed a long test run turn by turn. The model accounted for 14% of the time and the tools for another 14%. The remaining 71% was neither: it was the time between turns, spent handing the conversation from one step to the next.
A rerun of the same test, with the same agent, took 73% less time.
Show the numbers
| Measure | Model or work | Tools | Between turns |
|---|---|---|---|
| One long run | 14% | 14% | 71% |
| Median turn before | 25% | 75% | |
| Median turn after (4× faster) | 98% | 2% |
Figure: in one long run, 14% was model time, 14% tool time and 71% waiting. After we reworked how steps hand over, the median turn got four times faster, from 23.5 to 5.6 seconds, and waiting fell from 75% of a turn to 2%.
We reworked how a test moves from one step to the next, so that a turn no longer waits for the step around it, and kept every part of a run close to the conversation it reads and writes. A rerun of the same test, with the same agent, took 73% less time.
How good is Pip?
Quality is hard to measure, so we checked it two ways: with a QA agent on real tests, and with blind graders on a benchmark.
In the second half of October we ran a set of real tests from production on Pip, on the sites of a fashion store, a photo-products retailer and an airline, and had Claude Code check every result in a browser. The next day we merged that day’s fixes and ran the same tests again.
v0.7
v0.10
Show the numbers
| Day | Version | Works | Partly | Broken | Rightly no change |
|---|---|---|---|---|---|
| Day 1 | 0.7 | 6 | 6 | 0 | 1 |
| Day 2 | 0.10 | 9 | 3 | 0 | 1 |
Figure: the same set of real tests on two consecutive days. On the first day 46% worked, 46% partly worked, none were broken, and 8% rightly made no change because a bot wall blocked the page. On the second day 69% worked, 23% partly worked, none were broken, and 8% again rightly made no change.
For the benchmark, we took five real tests, ran each three times, and had blind graders check the results on fresh browser visits. Pip’s first drafts scored 0.90. After one round of review feedback they scored 1.00. A test took 4.4 minutes on average.
Compared with the previous variation agent, the clearest difference is cost. A test built by Pip costs about a sixteenth as much and takes about 30% less time, with 92 to 98% of its prompt served from the cache, where the previous agent left most of its tokens uncached. That agent scored 46 of 100 on its own eval set, which is not directly comparable with Pip’s benchmark.
Show the numbers
| Measure | Previous agent | Pip |
|---|---|---|
| Cost per test | 100% | 6% |
| Time per test | 100% | 71% |
Figure: Pip on its benchmark against the previous variation agent on pagent’s eval set. Cost per test: 6% of before. Time per test: 71% of before. The test sets differ, so treat both numbers as rough.
Our samples are small, often one or two runs per side, and the unchanged harness varies by about 9% in cost and 18% in time between identical runs. Hard cases cost much more: a test that needed a dozen generated product mockups cost up to 16 times an average test and took up to 48 minutes. And “it ran without an error” says very little. As our QA notes put it: “ok: true only means no exception was thrown, not that the change is visible.”
Pip moves into pagent
At the end of October we froze Pip at version 0.15 and moved it into pagent. In pagent’s staging environment, a build including a round of review costs a few percent of what the specialised agents spent on a test, at 80 to 90% cache rate.
Staging also gave us our favourite story. In one run the browser was unavailable, so Pip built a test blind. It did what a sensible person might do: it guessed the markup, submitted the change, and said it would “flag the verification limitation for review”. Its change looked for a button labelled “Add to cart”. The test shop’s button reads “Add to cauldron”. The variant did nothing.
Instead of adding a prompt line telling Pip never to build without a browser, we made a build without a browser fail in code. Our note on the ticket: “Enforce in code, not the prompt.”
What does Pip know about a website?
Instructions of 56 words do not mean Pip starts from nothing. Inside pagent, three extensions give it the customer’s context, each added with the same rules as everything else.
- The website and the test: who the customer is, what the website allows, the test’s goal, audience and the pages it runs on. Pip reads this once per run, so the prompt stays the same from turn to turn and the cache stays warm.
- The website’s knowledge: guidelines, brand voice and product facts, imported from documents and web pages or written in pagent. The part that matters for this test is in Pip’s briefing; the rest it reads when it needs it. Before it writes a claim like “free delivery above 50 euros”, it can check the claim against the website’s facts.
- The website’s design system: colours, type and spacing up front, the components on demand, so that a new badge or banner looks like the site’s own.
The knowledge is not static. When reviewers give feedback on a test, pagent suggests changes to the website’s knowledge. When a test ends with a decisive result, pagent suggests what the website’s knowledge should learn from it. Both show up as tracked changes with a comment, and a person accepts or rejects them. Feedback about one test’s flow stays with that test and does not become a rule for the whole site. When a suggestion contradicts an existing section, pagent asks which version is right instead of overwriting it.
Why not just use a coding agent like Claude Code?
We build pagent with Claude Code every day, and the QA runs in this post were done with it. It is an excellent agent for a repository you own. You can even drive pagent from Claude through our connector. But a general coding agent pointed at a customer’s website is missing most of what makes a test work:
- The output is not a code change to deploy. A coding agent edits files in a repository, and someone has to review, merge and release them. Pip produces pagent modifications, and it builds them for pagent’s SDK, which runs them on the live site without a release. The two are designed together. Pip prefers declarative changes, such as updating, inserting or moving an element, which the SDK keeps applying while a page hydrates and re-renders. The SDK holds the page back for a moment, so visitors never see the original flash before the variant. When a change needs a script, Pip writes it with its own undo and tracking: the SDK removes it as soon as it stops applying, for example when the visitor moves to another page, counts views and clicks on what it created, and tells a visitor who saw the change apart from one whose page never showed it. Triggers decide on which pages and steps a change runs, and the visitor’s consent decides what the SDK records. The customer’s engineering team does not have to be involved.
- It sees the page the way a visitor does. Pip’s browser opens the live site as a phone and as a desktop, from Europe, follows a journey across pages and gets past cookie dialogs. Pip’s browser, the review preview and pagent’s preview extension apply a change with the same code, so all three agree.
- It knows the website. Brand voice, product facts with a fact check, the design system, and what earlier tests and reviews taught pagent about this site.
- It works with people who are not engineers. A marketer reviews a preview, asks for changes in plain words, and Pip fixes the change in the same conversation that built it.
- It ends in a test. Every change becomes a variant in a Bayesian A/B test, and decisive results flow back into the website’s knowledge.
- It is measured on this one job. Every part of Pip was compared on real tests from production before it shipped.
What we learned
- Hand-offs lose context. One agent in one conversation beat a team of specialised agents passing work along.
- Instructions interact. Each of the nine prompt additions we tested fixed something and broke a check somewhere else.
- Add capability, context or evidence. A device option and one sentence of context got Pip to check phones, with no rule.
- Remove rediscovery. The best changes stopped the agent from finding out the same things again.
- Count turns. Every extra turn re-reads the whole conversation, so cost follows the number of turns and the cache rate.
- Boundaries belong in code. If something must always hold, do not ask the model to remember it.
- Check the grading too. Six of the 32 hill-climbing versions were fixes to our own grading.
- Profile the whole run. In one long run, 71% of the time was spent between turns, outside both the model and the tools.
- Grow one idea at a time. With one idea per change and a comparison for each, we could tell what helped whenever the effect was larger than the noise.
What’s next
Next, we want Pip to use test results more directly than through the website’s knowledge, so that a hypothesis that won on three pages informs the fourth without waiting for someone to write it down.
Unlike most agents, Pip can get a reward signal from Bayesian results on live traffic. That also makes it easy to learn the wrong thing. An agent that optimises for clicks on noisy tests could learn dark patterns: fake urgency, misleading buttons. We expect most finished experiments to teach Pip nothing at all, and the ones that do will go through the same rules as everything else: one idea, measured against the current Pip, with a person deciding.
We intend to keep Pip small, and the rules for contributors stay as they are: one idea per pull request, with evidence.
Frequently asked questions
What is Pip?
Pip is pagent's agent harness. It turns a test hypothesis into a working change on a customer's live website, checks the change in a real browser on phones and desktops, and fixes it from human review feedback, all in one continuous conversation per test.
What is an agent harness?
An agent harness is everything around the language model: storage, the conversation loop, the tools the model can call and the environments those tools run in. Pip's harness gives the model a browser, a way to submit a change, an image tool and access to attached files.
Why does Pip have so few instructions?
Rules we added to pagent's earlier agents often fixed one case and broke another. In a controlled test, nine targeted prompt additions all caused a regression somewhere. Pip's instructions stay at 56 words; its improvements add capability, context or evidence instead.
What is pi-durable and why does Pip use it?
pi-durable is an open source framework from Earendil for long-running, crash-safe agent conversations with extensions. Pip builds on it so that pagent only writes what is specific to building A/B tests: the tools, the browser and the review loop.
Is Pip cheaper than pagent's previous agents?
On a benchmark of five real pagent tests, a test built by Pip cost about a sixteenth of what pagent's previous variation agent cost on its eval set, and took about 30% less time. The test sets differ, so treat both numbers as rough. Pip keeps 92 to 98% of its prompt in the provider's cache, and its first drafts scored 0.90, reaching 1.00 after one round of review.
How is Pip different from a coding agent like Claude Code?
A coding agent edits a repository that someone then has to release. Pip produces pagent modifications that run on a customer's live site without a release, checks them in a browser as phones and desktops see them, uses the website's knowledge and design system, and turns every change into a variant in a Bayesian A/B test.
Does Pip learn from past tests?
Yes, through the website's knowledge in pagent. Review feedback and decisive test results become suggested changes to the website's guidelines and facts. A person accepts or rejects each suggestion, and Pip reads the accepted knowledge in every following test.
Does Pip replace human review in pagent?
No. Pip checks its own work in a browser, but a person approves every change before visitors see it. Review feedback goes back into the same conversation, so Pip fixes a change with everything it learned while building it.