Skip to content
Agentic Engineering Fieldbook

Research, Benchmarks, and Industry Developments

Observation date and maintenance

Section titled “Observation date and maintenance”

Observed on 2026-07-23.

Applicable sources: current public benchmark papers, benchmark project documentation, and evaluation articles for software-engineering agents, browsing agents, tool-agent-user interaction, and general assistant tasks.

Maintenance state: maintained Field Note. Revisit when a benchmark changes its task set, harness, contamination controls, scoring method, or when a new evidence class changes a durable Fieldbook claim.

Current observation

Section titled “Current observation”

Agent benchmarks are increasingly about systems, not isolated completions. Representative tasks now include resolving real software issues, browsing for hard-to-find information, interacting with tools and simulated users, and solving multi-step assistant tasks. This is progress: it pressures models and harnesses to deal with state, tools, uncertainty, and long-horizon work.

It also makes results harder to interpret. A benchmark score may depend on the base model, tool interface, prompts, scaffolding, retry budget, time limit, browser environment, package installation, evaluator, answer format, and leakage controls. A changed harness can alter the result even when the model stays the same. A better score can reflect better orchestration rather than a generally more reliable agent.

For the Fieldbook, the useful question is not “which model is winning today?” The useful question is “what evidence does this benchmark add about tasks, failure modes, harness design, and evaluation limits?”

Durable engineering implication

Section titled “Durable engineering implication”

Use public benchmarks as external probes, not release gates.

A benchmark can suggest that a class of tasks is becoming feasible. SWE-bench and SWE-bench Verified are relevant to coding agents because they use real software issues and test-based outcomes. τ-bench is relevant to tool-using agents because it includes simulated user interaction and domain tools. GAIA and BrowseComp are relevant to research and browsing agents because they stress multi-step information seeking, tool use, and evidence gathering.

None of those benchmarks proves that your product works. Your local system may have different repositories, permissions, APIs, documents, latency objectives, cost ceilings, privacy rules, escalation paths, and user expectations. A coding agent that performs well on benchmark issues can still mishandle your monorepo’s build system, secrets policy, migration process, or review culture. A browsing agent that answers benchmark questions can still cite stale sources in your regulated domain.

When a benchmark result matters, read the harness before reading the leaderboard. Ask what the agent could see, which tools it could use, how many attempts it had, how success was scored, whether tasks were filtered, whether answers or tests could leak, and whether failures were analyzed. Then decide whether the evidence changes a local evaluation suite, a design assumption, or a Fieldbook chapter.

What not to generalize

Section titled “What not to generalize”

Do not turn benchmark movement into a universal product promise. “Agents can resolve software issues” is too broad. A more responsible claim is bounded: under this benchmark’s issue distribution, environment, harness, budget, and scoring rule, this class of systems achieved this measured result.

Do not treat a public benchmark as a substitute for domain experts. Benchmark tasks often simplify acceptance, data access, policy interpretation, user preference, and organizational accountability.

Do not ignore negative evidence. Failure traces, abstentions, tool errors, budget exhaustion, hallucinated evidence, and benchmark contamination concerns are often more useful for design than headline scores.

Operational checklist

Section titled “Operational checklist”
  • Record benchmark version, task split, harness, tools, model configuration, retry budget, time budget, evaluator, and scoring rule before citing a result.
  • Distinguish model improvement from scaffolding, retrieval, browser, tool, or evaluator improvement.
  • Check whether benchmark tasks resemble your workload’s data, authority, consequence, latency, and success criteria.
  • Look for contamination, memorization, answer leakage, hidden filtering, and evaluator drift.
  • Convert relevant benchmark failures into local evaluation tasks when they match your product risk.
  • Never use a public leaderboard as the sole evidence for a production release.
  • Prefer dated Field Notes for volatile benchmark or industry developments; move material into core chapters only when it changes durable guidance.

References and how they are used

Section titled “References and how they are used”