Agent performance and virtual clock

TL;DR

• We run live-web agent benchmarks repeatably. Across repeated batches of the same 12 AssistantBench tasks, the share of requests served from cache climbed from 55% to about 83%, so a second run mostly does not touch the network.
• We cache Google searches. Searches go to a search API instead of google.com, which showed a captcha in 57 of 60 runs through our basic proxy system. Answers are cached on an exact key, because a “close enough” search was the wrong answer almost every time we tested it.
• A virtual clock measures the agent’s real-time runtime. Every cache hit is charged the time the real fetch took, so a warm cache makes a run cheaper, not the agent faster. Virtual time ran a median 1.05× to 1.15× real time.
• We cache static pages and assets so they are not fetched again. 46,690 entries (1.96 GiB) were replayed more than 160,000 times. Showing the agent its own clock and budget, with a larger thinking budget, took a 300-second run from 0 to 6 correct of 24: the same as a 900-second run, at under a third of the spend.

Live-web benchmarks are slow, expensive and unrepeatable. The same task run twice reads different pages, hits different captchas and waits on different servers. We now cache every page and every Google search the agent makes. A virtual clock then charges each cache hit the time the real fetch took. So a warm cache makes a run cheaper and faster to execute, but it does not make the agent look faster.

This post explains how the cache and the clock work and which traps we hit. It also reports what we learned about guessing searches ahead of time. The short answer to that: you mostly cannot, and serving a “close enough” search is wrong almost every time.

The worked example: AssistantBench

AssistantBench has 214 tasks: 33 validation tasks with answers and 181 test tasks.¹ They are realistic, closed-answer questions that take a person several minutes.² Here is one validation task:

Based on recent years (2020-2023), how likely am I to hit a rainy day (at least 0.5mm of precipitation) if I travel to Seattle during the first week of September?

The gold answer is 14.2. The gold evidence is four Weather Underground monthly history pages for the Seattle–Tacoma airport station, for September 2020, 2021, 2022 and 2023.¹

To answer it, an agent searches Google, opens one of those pages, waits for a heavy weather site to render, and repeats this for four years. On the live web, every one of those steps changes between runs. Google may answer with a captcha, the weather site may be slow, and the search results may reorder.

Our batches did not include this task. They ran 12 other validation tasks. The cache holds no Seattle history page, but it does hold 13 Weather Underground history pages for Chicago from another task. Those were fetched in 476–1,269 ms and then replayed up to 6 times each.³ The Seattle task is still the clearest way to show the idea.

What gets cached

The cache is a small server with a sqlite index. Bodies are stored on local disk and also written to S3. It never fetches anything itself. The browser’s request handler fetches through a proxy and then asks the cache to store the response.

The key is a SHA-256 of METHOD URL. Each entry has a class, and the class sets how long it lives:

Eviction. When the disk copy grows past 2 GiB, the server evicts the least-recently-used entries until it is back under the limit. It does this on every write, and also on demand.

Write-through to S3. Every write also goes to S3, as a new object per put, so a fresh machine can warm from S3. An S3 error other than “no such key” returns a 503, and the client treats that as a miss. A broken bucket costs speed, not correctness.

What is never stored:

• 5xx and 429 responses
• 206 and 304 responses
• responses marked private or no-store
• responses that vary on cookie or authorization
• requests that carry Authorization or Range
• bodies over 8 MiB

set-cookie is stripped from a replayed response. Media and EventSource requests are dropped altogether.

Egress. A miss goes out through our basic proxy system, then the basic proxy system again, then a separate proxy route. Government hosts go only through that separate route, never through the basic proxy system. Nothing is fetched from our own IPs.

Here is the cache as it stood on 2026-10-06:³

Assets are 92% of entries and 94% of bytes. A benchmark cache is mostly a cache of JavaScript and images.

The virtual clock

The rule is simple: virtual time = real time + Σ (charged − real) for every request the agent waited on.

• A live fetch is charged what it really took.
• A hit is charged the fetch time stored with the entry, not the 2 ms the disk took. So a replayed slow page still costs the agent its slowness.
• Model time stays real. In C-001, model time was 58% of run time (1,856 of 3,201 s).

Four details keep that rule honest.

The 3 s cap. An answered request is charged at most 3 s, whether it is live or replayed. The cache still records what it really took. The cap is about the basic proxy system’s p99 for a page. Its assets reach 19 s, and without a cap one slow proxy hop would decide a run.

Burst clamping. A page load fires dozens of requests in parallel. Summing their diffs would charge a parallel load as if it were serial. So within a burst, main-frame documents are summed. Of the sub-requests, only the largest positive diff and the largest negative diff count. Then comes the clamp: a burst can never take back more time than the refunded intervals that have actually settled cover. A replay can make the clock run slower than reality, but it can never make the clock run backwards past what really happened.

The lower-bound latch. Deadline checks use a lower bound: the smallest virtual time the run can still end up at. Once that passes the deadline, the stop latches, even if a later refund would pull the estimate back. There is also a real-time backstop at 3× the virtual deadline. A virtual stop is scored as a deadline stop, not as a crash.

Google results pages are charged a fixed 1.5 s, whether live or cached.

In C-001, virtual time ran a median 1.12× real. Across the later batches, the median ratio stayed between 1.05× and 1.15×.³

The control lever: the agent sees its clock

A clock the agent cannot see only punishes it. So every tool result now ends with a line like this:

[time: 212s used of 300s, 88s left; spend: $0.061 of $0.10]

The task prompt states the same budget up front. At 80% of either budget, the agent is told to answer. After the cutoff, any tool call returns “Time is up…” or “Budget is up…” instead of doing anything.

There are two subtleties.

• The refusal latches. Once a tool has refused, it keeps refusing. A spend estimate can fall back under the cutoff when a cheaper call settles, and the refusal has already told the agent no further tool call will run.
• The spend cutoff counts the model call in flight. Spend is the sum of priced calls plus the cost of the last call. Otherwise, the call that crosses the budget would be free.

The time cutoff waits on the lower bound, not the current estimate, for the same reason as the deadline.

What it did, on 12 tasks × 2 reps with glm-5.3-flash (fixtures: none, these are live sites through the cache):

At a third of the time and under a third of the spend, C-005 matched C-002’s 6 correct. All of these are small samples. One task flipping changes a row.

The SERP API: replacing google.com

Through our basic proxy system, google.com showed a captcha in 57 of 60 C-009 single-pass runs. So a search the page makes for google.<tld>/search?q=… is now answered by a managed scraping service’s Google Search scraper. The service has its own IP pool, and a query costs about $0.0035. The page gets a plain results page built from the JSON, and the agent clicks through to real sites.

• The key is SERP-API v1 | gl | hl | start | lower(q). Case and spacing do not split the cache, but nothing else is normalised. The section below explains why.
• Filtered searches fall back to google.com. These include images, news, date ranges, language and country filters, num, safe, the as_* advanced-search fields, and any udm other than 14. The JSON would not represent what was asked.
• Links are http(s) only. A javascript: or data: link in the provider’s JSON is dropped.
• The locale is kept. The results page’s own search box carries hl and gl, so the next search is asked in the same language and country.
• Empty results are not stored. Identical searches already in flight are deduplicated.

The SERP API is slow. Over its 139 cached answers, a fetch took a median 16.9 s (p90 31.2 s, max 45.5 s).³ The clock charges 1.5 s, which is what a Google page costs a person. A provider’s queueing is not the agent’s time.

Two traps

No egress may offer deflate

Playwright’s request context cannot read a raw-deflate body. It fails with “incorrect header check”. We hit this on a county government site in C-001. Every fetch the handler makes now sends accept-encoding: gzip, br.

• Fetches made through Playwright’s request context were fixed first.
• The basic proxy system’s route.fetch path offered deflate too, and needed its own fix.

A browser test pins it: it reads a page through the basic proxy system’s path and checks that the request never offers deflate.

Redirects must follow on the correct egress

The first version fulfilled a 302 straight to the browser. The browser then followed it internally, so our route handler never saw the next hops. We verified this locally: the server saw /start, /mid, /end while the route saw only /start. Each hop must be routed again because the next host may be a government site that needs the separate route.

The fix has two parts.

Navigation. A 3xx on a navigation is answered with a tiny page that calls location.replace(next), so the next hop is a fresh, routed request.

Sub-requests. For these, the handler follows the redirect itself:

• up to 10 hops, each on the egress its host needs
• credentials only to the same origin
• a 303, or a 301/302 after a POST, becomes a GET
• a 307 or 308 keeps the method and body
• a cross-origin POST 307 on a navigation is refused, rather than showing another origin’s page at the first URL

The tests that pin this check that the handler:

• answers a redirected navigation with a page that replaces itself, so the next hop is routed again
• follows a sub-request’s redirect itself, sending a government hop through the separate route without the page’s cookies
• strips the page’s credentials from a hop, and turns a 303 into a GET
• sends the body on a 307 and none on a 303, and keeps authorization on the page’s own origin
• never shows another origin’s page at the first URL: a cross-origin POST 307 navigation is refused
• does not bring a body back after a 303 dropped it

Automated code review found 12, then 11, then 5 defects across the rounds on this code. Before the C-001 re-run, 831 of 3,859 cached entries that the new rules refuse were purged.

A trap we have not fixed yet: a captcha redirect is cached

The code says “a captcha is never stored”, and the captcha page itself is never stored: the cache holds no google.com/sorry entry. But Google’s 302 that points to the captcha is stored as an ordinary serp entry, for a day. Of the 759 cached serp entries, 700 are that redirect, and they were replayed 139 times. Only 59 are real results pages.³ Each replay sends the agent straight to a captcha without even trying Google again. The fix is not to store a 3xx whose location is /sorry/. Until it ships and is re-measured, we do not claim the cache never hands back a captcha.

How well does the cache hit?

Hit rate is hits ÷ (hits + stores), per batch:³

The rate climbed from 55% to about 83% as the same 12 tasks were run again. In C-009 single, 27,227 requests were hits and 5,598 were stored. That is what the cache is for: the second run of a task mostly does not touch the network.

The SERP API hit far less often. Across three C-010 processes, 120 searches made 13 cache hits.³ That leads to the next question.

Can we guess the searches?

If the agent’s searches were predictable, we could prefetch them, or answer a near-miss from a neighbour. We checked both offline, on every Google search in the run traces: 1,182 searches in 370 runs, 942 distinct strings.³

One caveat first. 375 of the 379 runs used glm-5.3-flash. Only 4 used deepseek-v4.1-flash, so we could not say anything about overlap across models.

Exact repeats are rare. For each search the first time it appears in a run, we asked whether an earlier run of the same task had already asked it. Each normalisation level below adds to the one above:

Two runs of the same task share little. The mean Jaccard overlap of their search sets is 0.039 with our key, and 0.056 at the loosest level, over 5,355 pairs of runs. Per task, it ranges from 0.004 to 0.143. The first search is the most repeatable: it matched an earlier run’s first search in 145 of 354 runs. The most repeated search, “fubo management team site:fubo.tv”, appeared in 27 runs.

Loosening the key breaks it. We define two searches as equivalent when their first result is the same normalised URL and at least 4 of their top-5 URLs match. Then we checked this on the 139 real SERP-API result sets:

• Every collision that a looser key would create was not equivalent: 2 from punctuation and quotes, 3 from word order, 4 from dropping site:.
• Quoting “May 2020” changed all five top results.
• site:netflix.com and site:justwatch.com with the same words shared none.
• Serving a near-miss by word overlap was wrong almost every time:

The lesson is that a search the agent tweaked was tweaked on purpose. So we keep the exact key (case and spaces only). A cache that answers a different question than the one asked is not a cache. It is a fixture.

One caveat: we have not measured the noise floor, that is, how often the same query asked twice fails our equivalence test. If Google itself is unstable at the top-5, some of the “wrong” pairs above would be wrong even with an exact match.

Cold vs warm: did the scores move?

We have not measured this. The batches above ran one after another against the same cache, so every batch after C-001 ran warm, and we have not run the same tasks against an empty cache for comparison. So we do not claim that the scores are the same cold and warm. The clock is designed to make that true, but the claim needs a run.

Prior art

• WebArena sidesteps the live web. It ships self-hostable copies of four kinds of site (shop, forum, GitLab, CMS), so every run sees the same data.⁴ VisualWebArena adds 910 visually grounded tasks on the same idea.⁵
• Mind2Web records snapshots of real sites with the full interaction trace, so tasks replay offline. In exchange, the agent is graded against what the page was at collection time.⁶
• WARC replay (Webrecorder’s pywb) is the archivists’ version: record HTTP exchanges into WARC files and replay them with high fidelity, or act as a recording proxy to the live web.⁷
• BrowseComp keeps the live web but makes answers short and easy to verify, so grading is stable even when the web is not.⁸ BrowseComp-Plus went the other way. It replaced live search with a fixed corpus of about 100k documents, to make comparisons fair and repeatable.⁹

Our cache sits between these. The pages are real and fetched live the first time. After that they are frozen for days, not forever, and the time they took is kept. Snapshot benchmarks freeze the page and lose the clock. Live benchmarks keep the clock and lose repeatability. We try to keep both.