Cloning [part of] the world wide web for faster agent testing

TL;DR

• We build replicas of real SaaS sites (a CRM, a to-do app, ad managers, a spreadsheet, a calendar, a shop admin, a help desk) and score each one at three levels, pixel, DOM and API, against recordings of the real site.
• Our CRM replica replays 107/107 recorded API scenarios cleanly in both directions (775 steps), and the recordings catch 46/46 bugs we planted on purpose.
• Our to-do replica replays 38/38 API scenarios, catches 26/26 planted bugs, and reproduces 76.4% of the real page’s DOM hooks at the exact same path, with 95.4% of computed styles matching.
• The obvious pixel metric is useless: a blank white page scored higher than our clone. We score “ink agreement” instead, which a blank page cannot beat.
• The replicas fail on purpose (rate limits, lost saves, expired sessions), and tests grade the agent by what actually changed on the site, not by what the agent says it did.

Some of the websites we created:

• A CRM
• A to-do list
• A calendar
• A help desk
• A shop admin

Pixel-perfect. DOM-perfect. API-perfect.

We test web agents on replicas of real websites: a CRM, a to-do app, ad managers, a spreadsheet, a calendar, a shop admin, a help desk. A pass on the replica should mean a pass on the real site. That holds only if the replica is faithful where the agent looks, and different agents look at different layers. So we score every replica at three levels, each against a recording of the real site:

The headline numbers, measured offline from recordings:

• hubspot-sim: 107/107 API scenarios clean in both directions, over 775 request/response steps, and 46/46 deliberate bugs caught by the recordings.¹
• todoist-sim: 38/38 API scenarios clean both ways (165 steps), 26/26 deliberate bugs caught, and 76.4% DOM hook coverage (559 of 732 hooks on 7 real screens) at 95.4% computed-style match.²³
• Pixel: 12.5% ink agreement on todoist-sim. That is low, and the section on pixels explains why, and why the obvious pixel metric would have reported 93% instead.

This post is about the machinery behind those numbers: a recorder and a structural differ for APIs, a DOM signature for markup, a pixel metric that a blank page cannot beat, and a fault-injection surface so the replicas can fail the way real sites do. Where a score cannot be measured yet, we say so. At the end we compare the approach with WebArena, REAL, OSWorld and Mind2Web-Live.

Every number below was measured offline on 2026-10-06, from recordings and captures we already held. No call went to the real HubSpot, Todoist, Meta or X, and the spend was $0.⁴

The real sites behind these replicas (Todoist, HubSpot, X Ads, Meta Ads Manager and the others) are market leaders in their categories. We took inspiration from them because they are the sites agents are actually asked to use.

The simulators

What some of them look like (all data is fictional):

How we score each level

API-perfect: replay of real recordings

Our recorder runs each scenario against the real API and writes every request and response to a “golden” file. Replay then runs the same requests against the clone and diffs the two responses structurally.

Before diffing, ids are renamed in order of first sight (for example <ID3:digits:12>), and timestamps become <TSn:ms>. So two ids only have to be distinct and in the same places, not equal.

A reverse replay diffs in the other direction. It catches fields the clone sends that the real API never does.

The score is the scenarios that are clean in both directions, out of all recorded scenarios. We also give the number of request/response steps behind it.

The score has a limit: it only covers what was recorded. A clone can score 100% and still lack a whole feature nobody recorded. So every API score below comes with a “not mimicked” list.

DOM-perfect: the hook signature

Our DOM signature tool walks a rendered page and records every hooked element: one with an id, data-testid, aria-label or role. It identifies each one by its path of hooks, and records its bounding box and a fixed list of computed styles. A differ then compares the real signature with the clone’s. It reports:

• Coverage: the share of the real page’s hook paths that exist in the clone at exactly the same path. This is the headline number. An agent that says “click [data-testid=task-list-item] > [aria-label=Complete]” works only if that path exists.
• Loose coverage: the same, matched by the single strongest key (test id, then id, then aria-label, then role) instead of the full path.
• Style match: over hooks matched by path, the share of compared computed-style properties that are equal after normalisation (colours to rgba, px rounded).
• Bounding-box deltas: the median absolute x/y/width/height difference over matched hooks.
• Visible only: the same numbers, ignoring elements that are not visible in either page.

Pixel-perfect: ink agreement, not raw pixel match

Our first pixel metric was the obvious one: the share of pixels that are equal, or within 16 per channel (“tol16”). That metric turned out to be useless.

Most of a web app screen is white. So we also scored a flat white image against the real screenshot, as a floor. On the Todoist inbox screen, the blank image scored 97.6% tol16, and our clone scored 93.2%.⁹ A blank page “beat” the clone.

So the pixel score we report is ink agreement:

1. Take the real screenshot’s modal colour as the background.
2. A pixel is ink if it differs from that background by more than 16 on any channel, in either image.
3. The score is the share of ink pixels on which the two images agree within 16.

A flat image scores 0 by construction. We also report a best-shift variant: the best vertical offset between 0 and 80 px is applied before scoring. This separates “everything is 51 px lower” from “everything is wrong”.¹⁰

The scores

“Not measured” means no recording of the real site exists for that level. It does not mean we believe the score is low or high. We have not estimated any of them. Recording an API needs a real account per vendor; a DOM or pixel score needs a capture of the real screens in a logged-in browser. Today only hubspot-sim and todoist-sim have the first, and only todoist-sim has the second.

Fixture results versus live-site results

Everything in the table is a fixture result: the clone is compared with a recording, not with today’s real site. The recordings date from 2026-09-23. If HubSpot or Todoist changed after that, the replay cannot see it.

Live-site results exist only from the recording rounds. Those are in the next section and are labelled as live.

hubspot-sim

Replay, measured 2026-10-06:

Every skip is a deployment test that runs only in the deployment environment.¹

Live-site work, from the recording rounds (2026-09-23; we did not repeat it):

• The recordings came from a real portal.
• 46/46 mutations killed. Each mutation is a deliberate bug put into the clone; it counts as killed if at least one golden goes red.
• 675 live requests in the explorer and walk rounds.
• 484 live requests in the verification round. Run 7 was the only clean pass, and it covered one walk of 40 steps.
• A census was taken before and after each live recording: 6 contacts and 12 companies throughout. The recordings left the real portal unchanged.

What the real API taught us that the docs did not:

• A contact has one primary company.
• A v4 association PUT sets labels rather than adding to them.
• Empty values sort lowest.
• Search lags writes. We measured an index lag of 12.4 s once.
• Merges enrich records asynchronously.

Proven scope. Only contacts and companies are proven. Deals, tickets, products, line items and projects are unproven: both our tokens get 403 on them.

Not mimicked: rate limits as the real portal sends them, async search indexing, auto-association by email domain, id allocation order, and scopes.

todoist-sim

API. Replay, measured 2026-10-06:²

Live-site work, from the recording rounds (2026-09-23):

• 26/26 mutations killed.
• A catalogue of 244 generated edge cases.
• The first live pass found 172 disagreements in 49 classes, over 438 ledgered requests.
• The real API allows 100 full syncs per 15 minutes and answers more with a LIMITS_REACHED 429. The clone does not reproduce that limit.

DOM. The real side is the 9 screens captured on 2026-09-23 in a logged-in browser. The capture is HTML, CSS and sync logs. We keep it out of version control because it holds account data.⁵ Seven of those screens are used for scoring.

For each screen, the captured HTML was re-rendered offline with scripts stripped, and the DOM signature tool was run on the result. The clone side was captured on 2026-10-06 from a local copy of the clone, logged in as a throwaway workspace. The capture made 0 requests to anything but the local clone.¹³

Coverage per screen, 2026-10-06 clone:

The largest missing hooks are the onboarding cards in the sidebar, such as “Get Todoist on mobile”. The real account showed them and the clone’s seed does not.³

The clone uses the vendor’s DOM ids, test ids and aria labels, because those are what a DOM agent keys on.

Pixels. Mean over the same 7 screens at 1440x900:¹⁴

Three things to read from this:

1. Raw pixel match is nearly the same as the blank page’s. It cannot rank clones.
2. Ink agreement is about 2% on six of the seven screens. The task-detail screen scores about 75%, because a dark modal overlay covers most of it and both images have it.
3. DOM and pixels moved in opposite directions. Coverage went up 2.8 points between the two clone versions, and raw pixel match went down 2.6 points. A clone tuned for one level is not tuned for the other.

Why the pixel score is low. We looked at the PNGs side by side:

• The clone adds a fixture banner and a workspace bar, about 51 px tall, so everything sits lower. The best shifts are 23 to 64 px.
• The real reference is a re-render with scripts stripped. Todoist virtualises its task rows, so the rows never drew. The real side shows an empty list plus “Finish your setup” and “Get Todoist on mobile” cards.
• The seed data differs from the account that was captured.

So 12.5% is a lower bound measured against an approximate reference. It is not a fair pixel score yet. A fair one needs the real pages captured with scripts running, from an account seeded to match the clone, and the clone captured with its fixture banner hidden.

Faults on purpose

A clone that only ever succeeds tests the happy path. The real site rate-limits you, loses a save, logs you out mid-edit, and covers the page with a consent dialog. So the clones can do those things on command.

Every sim has the same control surface: /_sim/reset, /_sim/seed, /_sim/state, /_sim/fault and /_sim/clock. These sit behind an admin header. A test arms a fault, runs the agent, then grades by reading /_sim/state — what actually changed — rather than what the agent said changed.

Stale refs are not injected by a sim. Our fake browser executor mints refs for each page generation. If the agent uses a ref from before the page changed, it refuses with STALE_REF: “That ref is from before the page changed. Take a new snapshot.” So stale refs are tested at the executor contract, one layer above the sites.

Consent banners are not injected by any sim either. We searched every sim for one and found none. Consent handling is tested in the browser executor against snapshots of real banners. One example is a Funding Choices dialog captured from brainyquote on 2026-10-01. Tested this way, banners are answered only when the banner is on an allowlist. No end-to-end run yet shows an agent getting past a consent banner on a site we control.

Clocks. Every .trade sim pins a clock. The default is 2026-10-05 09:00, and calendar-sim pins Monday 2026-10-05 09:00 in Asia/Jerusalem. helpdesk-sim advances the clock 1 s per write. Without a pinned clock, “next Tuesday” is a different date each run.

Per-visitor state. On hubspot-sim and todoist-sim, a cookie gives each browser a private copy of the seed, forked lazily, with at most 500 copies kept (least recently used). Named workspaces at /u/<name>/ are isolated from each other.

The four .trade sims do not have this yet. Their landing pages send every visitor to the same demo workspace, so one visitor’s edits show to the next. Per-visitor isolation for them is planned and not shipped as of 2026-10-06.

Four stories

Each of these is from our own notes, re-checked against the code for this post.

1. The versionless Graph path. Our first Graph bench refused requests without an API version. Meta does not: it serves a versionless path with 200, at the oldest supported version. The only clues are a facebook-api-version response header and old spellings like CONVERSIONS instead of the ODAX objective names.

Our clone was stricter than production, so it could never catch an agent that forgot the version. The bench now does what Meta does. Its specification opens: “It must not be lenient.”⁶

2. limit=100 meant no paging. The bench seeds 18 campaigns. A test asked for limit=100, so the response had no paging.next. A mutation that stripped paging could never turn it red.

With limit=2, the lock fires. The bench also puts the access token into paging.next, as Meta does. A test that drops it is caught.⁶

3. Twelve green proofs over the wrong error. Navigation tests in a fake reported NAVIGATION_FAILED where the real executor reports ORIGIN_NOT_ALLOWED. Twelve proofs and two mutation matrices passed, because they all agreed with the fake.

A fake proves your model of the other side, not the other side. This is why every API score above comes from a recording.¹⁵

4. Mirror the producer, not a namesake. A grader pinned a string produced by a helper function with the right-sounding name. On the page that test reads, that function never produces the string. The grader passed while checking nothing.

The rule we took from it: a clone’s assertion must point at the code that produces the text on screen.¹⁵

Prior art

Where this work differs. None of these scores a clone’s fidelity to the real site, level by level. None of the ones we read injects rate limits, lost saves or consent banners on purpose.

The self-hosted benchmarks (WebArena, WorkArena) get fidelity for free, because the app is the real app. They do not cover proprietary SaaS. The live benchmarks (WebVoyager, Mind2Web-Live) are faithful but cannot be controlled or graded by state. REAL is controlled, but does not claim fidelity.

We want both: clones of the proprietary sites agents are actually asked to use, scored against recordings of those sites.