Classic CRO vs continuous website experimentation: what is the difference?
Christopher, Founder @ pagentClassic conversion rate optimisation (CRO) is a discipline in which a specialist or a small team researches a website, writes hypotheses, and runs a small number of carefully planned tests. Continuous website experimentation is an operating model in which testing runs all the time, across many pages and audiences, and is part of how the website changes at all. Both use the same controlled experiment and the same statistics. They differ in cadence, in who owns the work, and in what one test costs to produce.
Neither is the better choice for every team. A site with little traffic and no weekly owner gets more from a focused CRO project, and a site with plenty of traffic and a backlog nobody works through is usually limited by production capacity. This article compares the two, lists the safeguards both need, and ends with a checklist and a decision guide.
The terms “classic CRO” and “continuous experimentation” are not standardised, and vendors and researchers use them differently, so the definitions here are ours.
What does classic CRO include?
Classic CRO usually runs as a project or a retainer. Someone audits analytics, session recordings and customer feedback, turns the findings into a prioritised list of hypotheses, and then tests the top of that list one at a time. Each test gets a design, a build by a developer or in a visual editor, manual QA, a fixed run time and a written analysis.
The strength of this model is the quality of each test. The hypotheses come from research, the changes are often large, and a specialist reads every result. The cost is throughput, because every step passes through the same few people. GrowthBook’s maturity guide describes this stage as running tests manually and puts it at “1 to 4 tests per month”.
What does continuous experimentation change?
Continuous experimentation treats testing as a standing process with no start or end date. There is always a queue of hypotheses, tests launch every week, results feed the next hypotheses, and the people who review and approve are named in advance.
The term comes from software product development. Fabijan and colleagues at Microsoft describe four stages of maturity, from “crawl” to “fly”, and in the last stage controlled experiments are the norm for every change, including small bug fixes (summary of the ICSE 2017 paper). At the far end of that scale, Booking.com runs “some 25,000 tests a year”, according to Stefan Thomke in Harvard Business Review. Very few websites need that volume, and we mention it only to show the upper end of the scale.
Show the numbers
| Programme | Tests | Source |
|---|---|---|
| Manual programme | 1 to 4 a month | GrowthBook |
| Platform-based programme | 5 to 100 a month | GrowthBook |
| Booking.com | about 2,100 a month (some 25,000 a year) | Harvard Business Review |
Figure: tests per month on a logarithmic scale. Manual programmes run 1 to 4 tests a month and platform-based programmes 5 to 100, according to GrowthBook. Booking.com runs some 25,000 tests a year, according to Harvard Business Review.
Is continuous experimentation just running more A/B tests?
Not on its own, because a team that triples its test count without changing anything else gets three times the QA load, tests that overlap on the same pages, and more false positives.
An operating model has three things that a higher test count lacks:
- Hypotheses are collected and ranked all the time, so the next test never waits for a research phase.
- Metrics, run times, stopping rules and approval steps are decided once for the programme and applied to every test.
- Every result, including the flat and negative ones, is recorded and used to write the next hypothesis.
Kohavi and colleagues at Microsoft add a warning about cheap tests. Once experimentation costs little, “it is easy to fall into the trap of answering many trivial questions”, and they point to the well-known case of a team testing 41 shades of blue.
How do the two models compare in practice?
| Classic CRO | Continuous experimentation | |
|---|---|---|
| Unit of work | A project or a quarterly roadmap | A standing queue with a weekly cadence |
| Source of hypotheses | Research phase by a specialist | Research plus a continuous flow from results, audiences and page data |
| Who builds variations | Developer or specialist, per test | A repeatable production process, increasingly assisted by AI |
| QA | Manual, per test | Standard checks on every variation, the same each time |
| Typical cadence | A few tests a month | Tests launch weekly, several run in parallel |
| Coverage | Highest-traffic pages and biggest ideas | Those plus the long tail of pages, segments and campaigns |
| Analysis | Written up per test by a specialist | Standard metrics and rules, reviewed in a regular meeting |
| Governance | Sign-off per project | Named reviewers and an approval step per test |
| Main risk | Too few tests to learn from | Weak hypotheses, overlapping tests, false positives |
| Main bottleneck | Specialist and developer time | Traffic and review capacity |
Trade-offs by traffic, team size and technical capacity
How much traffic do you need?
Traffic sets the upper limit on how many tests a website can run, and no operating model changes that. Evan Miller’s rule of thumb for sample size is n = 16σ²/δ² per variation, where σ² is the variance and δ the smallest effect you want to detect.
In a worked example with our own numbers, a page converts at 3% and you want to detect a rise to 3.3%. The variance is 0.03 × 0.97 = 0.0291, the effect is 0.003, and the rule gives about 52,000 visitors per variation, so roughly 104,000 for a test with one variation. If you only care about a rise to 3.6%, the same rule gives about 13,000 per variation.
Halving the effect you want to detect quadruples the visitors you need.
Show the numbers
| Effect to detect | Visitors per variation |
|---|---|
| Detect a rise from 3% to 3.3% | about 52,000 |
| Detect a rise from 3% to 3.6% | about 13,000 |
Figure: visitors needed per variation on a page that converts at 3%, with Evan Miller’s rule of thumb. Detecting a rise to 3.3% takes about 52,000 visitors per variation, and detecting a rise to 3.6% takes about 13,000.
A page with 20,000 visitors a month can therefore test large changes and little else, and it can run perhaps one test at a time, so continuous experimentation on that page means a queue of one. The model pays off when several pages or audiences each have enough traffic for their own tests.
What can a small team do?
A small team without spare engineering time is the usual case for classic CRO, because each test competes with everything else on the developer’s list. It is also the case in which a continuous model is hardest to run by hand, since hypotheses, variations and QA all need hours that the team does not have.
The realistic options are to stay with a few well-chosen tests, or to reduce the cost of producing each test so that the same people can review more of them. The second option only works if someone still has a fixed slot each week to review and decide.
What changes for an agency?
An agency runs the same process on many client websites, each with its own design system, approval chain and traffic level. Classic CRO scales with headcount in that setting. A continuous model needs separate queues and separate reviewers per client, and reporting that a client can read without a call.
Which safeguards apply to both?
Running more tests does not relax any statistical rule, and several problems grow with volume.
- One primary metric per test, chosen before launch. Secondary and guardrail metrics are there to catch harm.
- A stopping rule decided in advance. Evan Miller shows that checking a test repeatedly and stopping at the first significant reading can raise the false positive rate to 26.1% when the reported level is 5%. Fixed sample sizes, sequential designs and Bayesian methods all address this, as long as the rule is set before the test starts.
- A sample ratio check. Microsoft reports that about 6% of its A/B tests show a sample ratio mismatch, which means the split of visitors differs from the configured one, and advises not to trust such a test until the cause is found.
- Rules for overlapping tests. Two tests on the same page or the same visitors can interact. Kohavi and colleagues describe both prevention, by never assigning a user to two conflicting tests, and detection after the fact.
- A second look at winners. In the same paper, a team that tries five treatments raises its chance of a false positive from 2.5% to 12%. A re-test of important winners is the simplest protection.
- QA on real devices before launch. A variation that breaks on mobile produces a result, and the result is wrong.
- An honest record of flat results. At Microsoft, “only one third of the ideas tested” improved the metric they were built to improve. A programme that reports only winners is hiding most of what it learned.
Show the numbers
| Habit | Before | After | Source |
|---|---|---|---|
| Stopping at the first significant reading | Reported level: 5% | With repeated checks: 26.1% | Evan Miller |
| Trying several treatments | One treatment: 2.5% | Five treatments: 12% | Kohavi and colleagues |
Figure: false positive rates. Checking a test repeatedly and stopping at the first significant reading can raise the rate from the reported 5% to 26.1%, according to Evan Miller. Trying five treatments raises it from 2.5% to 12%, according to Kohavi and colleagues.
How do experimentation programmes fail?
These failures occur in both models, and a higher cadence makes each of them more expensive.
- Weak hypotheses. Tests without a reason behind them produce results nobody can use, whatever the outcome.
- Poor implementation QA. A variation that renders wrongly on one device measures the bug.
- Overlapping tests. Two tests on the same visitors without a rule for it make both results hard to read.
- Premature decisions. A test stopped at the first good reading is the most common source of winners that later disappear.
- No owner. A queue without a person who reviews and decides every week stops within a month or two.
Readiness checklist for continuous experimentation
Count how many of these are true for your website today.
- At least one page or funnel step has enough traffic to finish a test within about four weeks.
- More than one page or audience meets that bar.
- You have one agreed primary conversion metric, and it is tracked reliably.
- One person owns the test queue and has time for it every week.
- Reviewers for brand, legal and product are named and answer within days.
- You can change the website for a test without waiting for a release.
- Run time and stopping rules are written down and applied to every test.
- Results are stored in one place, including the tests that showed no effect.
With six or more, a continuous model is realistic. With three to five, a hybrid is the sensible step. With two or fewer, start with a focused CRO project and fix tracking and ownership first.
Which model fits your situation?
| Your situation | Suitable model | Why |
|---|---|---|
| Under about 20,000 sessions a month, or few conversions | Classic CRO, with research and large changes | Traffic allows only a few tests, so each one has to count |
| Enough traffic, but no tracking or no owner | Classic CRO project first | The basics have to work before volume helps |
| Enough traffic on several pages, 1 to 3 tests a month, a backlog that never shrinks | Hybrid | Production capacity is the limit |
| An existing testing platform and a planned roadmap, with pages and segments the roadmap never reaches | Hybrid | Keep the planned tests where they are and add capacity for the rest |
| A dedicated team, standard metrics, and tests already launching weekly | Continuous | The operating model is in place |
A hybrid usually looks like this in practice. The strategic tests, such as a new checkout or a pricing page, stay with the CRO lead and the existing platform, with full research and analysis. A continuous queue covers the pages, audiences and campaigns that never reach that roadmap. Both use the same metrics and the same stopping rules, and one person sees all running tests so that they do not overlap.
Which parts of the workflow can AI assist with?
The slow steps in classic CRO are production steps. Writing page-specific hypotheses, building a variation in the website’s design, and checking it on desktop and mobile take most of the hours, and they are the steps AI can now assist with.
This is the part of the workflow pagent covers. It writes hypotheses for a specific page, builds each variation as real code in the customer’s design system, checks it on desktop and mobile, and sends it to the team for review. You approve every test before it goes live, and reviewers can approve, request a revision or veto by email. Tests run against a control and are measured with Bayesian statistics. We described how the agent behind this works in Pip: the agent harness we built by taking things away.
Three decisions stay with the team in any setup we would recommend:
- which metric counts as success, and which guardrails must not move,
- whether a variation fits the brand and may go live,
- what a result means for the business and whether to ship the change.
Before you extend automation, ask for evidence on a small scale. Run it on a few pages, compare the variations it produces with what your team would have accepted, check that its results hold up on a re-test, and measure the review time it costs you. pagent runs beside an existing testing platform, so the planned tests can stay where they are, as we describe in our guide to Optimizely.
For the wider context, see how we approach conversion optimisation that ends in a measured change, how interface changes become test hypotheses in UI optimisation with pagent, and our earlier tips on improving conversion rate with generative AI.
Frequently asked questions
What is the difference between CRO and experimentation?
CRO is the goal and the discipline of raising a website's conversion rate, and it includes research, design and copy work as well as testing. Experimentation is the method of measuring a change against a control. Classic CRO uses experiments occasionally, and continuous experimentation makes them the default way a website changes.
Can a small e-commerce team run continuous experiments?
Yes, if traffic allows it and one person owns the queue every week. The team size matters less than the cost of producing each test and the time available for review.
How many tests a month count as continuous?
There is no agreed threshold. GrowthBook's guide places manual programmes at 1 to 4 tests a month and platform-based programmes at 5 to 100, which is a vendor's estimate and a useful rough scale. A more useful sign is that tests launch every week without a new project being set up.
What should stay manual in A/B testing?
The choice of metrics, the approval of each variation, and the decision to ship a winner. Production steps such as drafting hypotheses, building variations and running checks can be assisted.
How do you move from project-based CRO to a continuous process?
Keep the current roadmap, fix the rules for metrics and stopping, name the reviewers, and start a second queue on two or three pages with enough traffic. Extend it when the results hold up and the review load is manageable.