Overfitting is good for the agentic web: why we cheat in benchmarks

TL;DR

• We fitted tools to a benchmark’s question shapes on purpose. That is the “cheat” in the title: legitimate overfitting to the kind of question, not leaking answers. No tool holds a reference answer. Most questions in DeepSearchQA, Google DeepMind’s web-research benchmark, follow one pattern: take a table of entities, filter it, then pick, count or rank.
• So we built MCP-like search tools over a website’s data. The first one sits on a database of GPU specifications. Instead of browsing spec pages, the agent sends one typed query (filters, a sort, an aggregate such as average or maximum) and gets back matching rows, each with its source link.
• That GPU search tool kept its gains on questions written before it existed. With the cheap model glm-5.3-flash, browsing alone answered 12 of 15 held-out runs. With the tool it answered 15 of 15. The median number of agent actions went from 14 to 2, and the mean cost per task fell from $0.0141 to $0.0011.
• Where the site already has a clean API, a tool mostly saves money. On World Bank questions, plain HTTP access and typed tools were about equally accurate (17 of 18 runs each on held-out questions). The typed tools cost about a quarter as much.
• One tool fixed a site’s trap and created a new one. On a mutual-fund screener, a screener tool lifted the design questions from 3 of 5 to 5 of 5. The held-out questions stayed at 4 of 5: the tool’s own description misled the model on one of them.
• None of these scores is a leaderboard number. They answer one question: does the agent get better at work shaped like this? Below we say which tools transfer and which kinds of fitting would only game the test.

Why a benchmark is worth fitting to

DeepSearchQA is a benchmark of 900 hand-written research questions in 17 fields, published by Google DeepMind with Gemini Deep Research in December 2025.¹ Each question is a causal chain: you need the answer to one step before you can start the next. A response counts as fully correct only when its answer set matches the reference set exactly, with nothing missing and nothing extra. The benchmark does not prescribe tools. It grades the final answer, not how the agent found it.

Here is one of the questions:

Out of all the GeForce GPUs released before 2020 that have over 1 GB base memory, what is the BUS width of the card that was known for black screen or BSoD crashes?

The reference answer is 352 bit. The question has two halves. Which card was known for crashes is a discovery problem: you search, read contemporary coverage, and weigh reports. Coverage in late 2018 points to the GeForce RTX 2080 Ti. What is its bus width is a database lookup. A generic browsing agent treats both halves the same way: it searches, opens pages and reads them. That works for the first half. For the second half it is slow, and on questions that filter a whole catalogue it is fragile.

So the question we care about is not “how do we raise the score?” It is “what should a tool look like for each shape of question?” A benchmark of 900 questions is a large sample of the work people want agents to do. Fitting tools to its shapes is a reasonable way to decide what to build.

Overfitting is a known hazard for agent benchmarks. Kapoor et al. found that many agent benchmarks have inadequate holdout sets, and sometimes none at all, and that this has produced fragile agents that take shortcuts.² The answer is not to avoid fitting. It is to hold out data, so you can measure whether the fitting transferred. Teams that build agent tools describe the same practice: tune tool definitions on one set of evaluations and check them on a held-out set.³

Three kinds of overfitting

“Overfitting” covers three different things. Only one of them is our thesis.

Here is what we did not do and would not count. We did not memorise answers. No tool contains a fact taken from a reference answer. We did not filter the GPU catalogue by “352 bit” to find the card, since that leaks the answer into the search. We did not tune a tool after seeing a held-out question. The held-out questions were written and frozen before we built the tools.

The core pattern: an entity table, then filter, then pick

Read enough DeepSearchQA questions and one shape keeps returning:

1. Take a set of entities: countries, GPUs, funds, US states.
2. Filter it on several attributes.
3. Pick the top or bottom item, count the rest, average them, or list them.

The hard part for a browsing agent is not reading a page. It is reading every page that matters: all 2,824 GPUs, or all 217 economies, or every page of a paginated result. It also has to respect the defaults the site applies without saying so. A typed tool does that part in code and gives the model only the rows it asked for.

Not every question fits this shape. Here is how four from the dataset sort:

Example 1: a GPU database tool

The benchmark question (DeepSearchQA)

Out of all the GeForce GPUs released before 2020 that have over 1 GB base memory, what is the BUS width of the card that was known for black screen or BSoD crashes?

Reference answer: 352 bit

The tool the agent gets instead of browsing:

{
  "name": "gpu_query",
  "parameters": {
    "filters":   [{ "field": "<one of 17 fields>",
                    "op":    "eq|ne|gt|gte|lt|lte|in|contains|starts_with|…",
                    "value": "..." }],
    "sort_by":   "<field>",
    "order":     "asc|desc",
    "fields":    ["<field>", "..."],
    "limit":     50,
    "aggregate": { "fn": "avg|sum|min|max|count", "field": "<field>" }
  }
}

Building it

We started from an open-source dataset of 2,824 GPUs from eight vendors, built from a public GPU specification site.⁴ Each row has the fields a hardware question asks about: vendor, product line, architecture, release date, memory size, type and bus width, bandwidth, TDP, die size, transistor count, FP16 and FP32 throughput. Each row also has the URL of its source page. We did not crawl anything ourselves.

The tool is the one shown above: a single function with schema-validated filters, an optional sort and an optional aggregate over all matching rows.

Three design choices matter more than the schema:

• The aggregate runs over every matched row, not the page the model sees. So “the average TDP of the series” can never be the average of the first page.
• Nulls sort last, and the result says how many were missing. A GPU with no published TDP must not win “lowest TDP”.
• Every row carries its source URL. The answer can cite where each value came from.

Here is a real call, for “NVIDIA data-center GPUs released 2022–2024 with at least 80 GB of HBM: which has the highest bandwidth?”:

{ "filters": [
    { "field": "manufacturer",  "op": "eq",          "value": "NVIDIA" },
    { "field": "generation",    "op": "starts_with", "value": "Server" },
    { "field": "release_date",  "op": "gte",         "value": "2022" },
    { "field": "release_date",  "op": "lt",          "value": "2025" },
    { "field": "memory_size_gb","op": "gte",         "value": 80 },
    { "field": "memory_type",   "op": "contains",    "value": "HBM" } ],
  "sort_by": "memory_bandwidth_gb_s", "limit": 3 }

{ "matched": 14,
  "rows": [
    { "name": "H200 NVL", "release_date": "2024-11-18", "memory_size_gb": 141,
      "memory_type": "HBM3e", "memory_bandwidth_gb_s": 4890, "source_url": "…" },
    { "name": "H200 SXM 141 GB", "release_date": "2024-11-18", "memory_size_gb": 141,
      "memory_type": "HBM3e", "memory_bandwidth_gb_s": 4890, "source_url": "…" },
    { "name": "B200", "...": "..." } ],
  "truncated": true }

Two cards tie. The question says to list all tied cards, and the reference answer lists both.

Measuring it

We wrote ten questions before we built the site or the tool. Five are design questions, the ones we had in mind while designing the tool. One of them is the DeepSearchQA question above. Its discovery half was given as a stated fact, so it measures only the lookup. The other five are held-out questions with shapes the design set does not have: an average over a series, a list with two conditions, a smallest-die pick, and a maximum over every GPU released before 2015. We computed each reference answer from the dataset, with its source rows.

Both arms used the same model (glm-5.3-flash), the same prompt and the same budget of 24 tool calls. The browsing arm could open a spec site rendered from the same rows: a filterable list of 100 rows per page, linked to one page per GPU. It could also run Python. The tooled arm had all of that, plus gpu_query. Each question ran three times per arm, on 2026-10-10.

What the misses were:

• All three held-out browsing misses are one question: “which GPU released before 2015 has the most transistors?” Answering it means reading the transistor count of about two thousand GPUs. The list pages do not show that field, so the agent had to open GPU pages one by one. It ran out of calls all three times. With the tool it is one call.
• The design-set browsing miss had the right answer in the wrong format. It joined the two tied cards with a comma instead of the separator we asked for, and our strict grader failed it. We report it as graded.
• On the other eight questions, browsing was as accurate as the tool, at 10 to 13 times the cost.

The tool is not fitted to its design questions. It is fitted to a shape that held-out questions share. And the gap between the arms is largest on the question that the browsing arm cannot finish within its budget.

Example 2: World Bank tools, where the site already has an API

Five DeepSearchQA questions need World Bank indicators.⁵ We ran them against a replica of the World Bank Indicators API, built from captured responses, so every run reads the same data. Four of the five also need a fact from another source, such as a list of countries that banned corporal punishment. We gave that fact to both arms as a stated premise, so these runs measure only the World Bank half.

The generic arm had raw HTTP access to the API and Python. The typed arm also had three tools: list the economies by region or income group with aggregates removed, fetch a whole series across all pages, and rank a year. We then wrote six held-out questions on the same eight indicators, such as “which country in the World Bank’s South Asia region grew fastest in 2021?” and “in which years did Greece’s GDP shrink?”. Three runs per question and arm, glm-5.3-flash, 2026-10-10.

On held-out questions, the typed tools made the agent cheaper, not more accurate. The API is already well structured. With Python, a cheap model pages through it correctly almost every time.

The interesting misses were on one design question. In the World Bank’s 2022 data, France and the Netherlands tie for fifth-highest infant mortality among large European countries. Break the tie one way and the answer is the Netherlands, which is the reference answer. Break it the other way and the answer is the United Kingdom. The generic arm answered the United Kingdom twice. The typed arm answered it once. That is a fact about the question as much as about the agent, and no tool can fix it.

Example 3: a fund screener, where the tool made a new trap

A mutual-fund screener has the traps that make agents report wrong answers with no error. It hides closed funds by default, sorts by a column the question did not name, and shows 100 rows per page. We built a replica of one from captured data and a screener tool that queries the full data set with no hidden defaults. We then measured browsing against the tool on five design questions and five held-out questions, one run each, with glm-5.3-flash.

The tool cut about nine calls to one and cost about 25 times less. It also lifted the design set. It did not lift the held-out set.

The browsing misses were all runs where the model stopped without an answer. None fell for a trap: whenever it did answer, it had found the hidden criteria or walked every page. The tooled miss is the instructive one. The tool’s description said it had “no hidden defaults”. On a held-out question about funds “listed by default”, the model read that as open or closed to new investors. It answered 245 instead of 174. The tool removed the site’s trap and its description added a new one.

This mirrors a published example from tool building: a model kept adding the current year to search queries until the tool description was changed.³ Tool descriptions behave like code. They need regression tests on questions the author did not write them for.

With five questions per cell and one run each, these numbers show a direction and nothing more. A 4 / 5 has a 95% interval of about 38–96%.

What transferred and what did not

An earlier experiment of ours points the same way. A cheap model wrote extractor configurations for site tools from one example page per site. They passed 29 of 40 checks on the sites they were written for, and 2 of 7 on held-out pages of those sites.⁶ Paths fitted to one example page did not generalise. In this post’s three examples, a tool fitted to a shape of question did.

On seven more tools built the same way, the tools won overall: 72 of 87 runs right against 59 of 87 for browsing, at a seventh of the cost, with a median run of 8 seconds against 31. The gains are not even. Where no single page holds the answer (NBA seasons, exoplanets), browsing collapsed and the tool was near perfect. Where our test site already had a page per state or per year (mortality, crashes, drugs), browsing was already right and the tool mostly made it cheaper and faster. Our first run of this showed no gain at all; the cause was not the tools. Some providers serving the model were silently dropping the tool’s filter arguments, so the agent looped on unfiltered results until it hit the step limit.

Here is every DeepSearchQA question we built and measured a tool for, with the same model (glm) on both sides:

Counts are correct runs, strict grading, three runs per question. Each row counts its design question and the held-out questions written before the tool was built; the GPU row is the held-out set only. Browsing ran against offline copies of each site with a page for every year, cause or state, which is kinder to browsing than the real sources are. Both sides used the same provider exclusion.

† Most of the tool’s misses here are the right entity with extra text on the answer line (a bank’s city, a crater’s diameter), which strict grading marks wrong. On #136 the tool found Swift, Voltaire and Lucian but missed Brontë and Shakespeare, which the USGS files as astronaut-named Moon features rather than as tributes to the writers.

Software-engineering agents show a similar pattern. An agent-computer interface designed for the task moves scores substantially.⁷ And published scores fall sharply on harder or private question sets.⁸ Expect held-out numbers to be lower than the numbers you designed against, and report both.

Honesty section

• The tooled scores are not leaderboard numbers. We ran the GPU and World Bank questions against data we hold. Discovery steps were given as stated premises. The graders compare exact answer sets. A DeepSearchQA leaderboard run is open-web, end to end. Our numbers say whether a tool helps an agent on work of a given shape. They do not say where the agent would place on that leaderboard.
• Samples are small. Five to six questions per set, three runs each, one model. The fund-screener results come from one run per question.
• Our browsing arm was kinder than the real web. The GPU site and the World Bank API were served locally from the same rows the tools use: no captchas, no slow pages, no layout changes. On the live web the gap would probably be larger, but we did not measure that.
• We have not measured the discovery half. The “known for crashes” step, which is exactly where search is still needed, was not part of these runs.

What we build next

The tools here are hand-written. We have not yet tested whether an agent can write the next one itself: observe a site, find its catalogue or API, and produce a typed query tool with held-out tests. That is the experiment we want to run, measured the same way: tools built from one set of questions, frozen, and checked on questions they never saw.