Evaluation: How Do We Know the System Works?

“The model is capable,” “the demo looked good,” and “users seem satisfied” are different claims. None, by itself, establishes that a particular AI-native system works reliably for its intended task.

The deployed behavior comes from more than a model. Instructions, context, retrieval, tools, permissions, control flow, application code, and the execution environment all influence the result. Evaluation must therefore describe the system and the work it is expected to perform, not only a model score.

Thesis

Evaluation turns intended system behavior into inspectable evidence. It begins by defining success, acceptable variation, and unacceptable failure for representative work; then it measures the model and the surrounding system together across outputs, execution, and outcomes.

Evaluation is not a single benchmark or launch gate. It is an engineering feedback loop:

intended outcome → explicit criteria → representative tasks → observed behavior
       ↑                                                    ↓
       └──────── diagnosis, correction, and new cases ──────┘

The loop does not replace product judgment. It makes current judgment explicit enough to test, challenge, and revise.

Start with the claim

Before choosing a metric, state what the system is supposed to do and for whom.

“Produces good answers” is too vague. Compare three more useful claims:

“For supported return requests, the assistant identifies the applicable policy and gives the correct next step.”
“For a defined set of repositories, the coding agent produces a patch that passes required tests without modifying excluded paths.”
“For research questions within scope, the system distinguishes sourced conclusions from unsupported or conflicting evidence.”

Each claim implies different tasks, evidence, and consequences. The relevant unit may be a response, a completed artifact, a tool action, a changed environment, or a later business outcome.

Evaluation is therefore close to an executable or operational specification. It exposes unresolved requirements early: what does “supported” mean, which tests are required, which sources count, what variation is acceptable, and what happens when the system cannot complete the task?

Benchmark, model, and system evaluation

These levels answer different questions.

Level	Question	What it can support	What it cannot establish alone
Public benchmark	How do models or systems compare under a standardized procedure?	Capability screening and broad comparison	Performance in our context, tools, users, and risk boundaries
Model or prompt evaluation	How does a model behave on a defined input–output task?	Model and prompt selection for a bounded capability	Correctness of retrieval, tools, permissions, or side effects
Component evaluation	Does one retriever, router, tool, or policy check perform its contract?	Diagnosis and local regression detection	End-to-end task success
System evaluation	Does the assembled product complete representative work within constraints?	Release evidence and architecture comparison	Every future production condition
Production outcome evaluation	Does real use create the intended effect without unacceptable harm?	External validity and new failure discovery	Controlled causal attribution without careful design

HELM demonstrated why even model evaluation benefits from multiple scenarios and metrics: systems can expose different trade-offs rather than one model dominating every concern (Bommasani et al., 2023). For an application, the case for multiple views is stronger because the model is only one dependency.

The basic units of an evaluation

Use distinct terms so that one percentage does not hide what was actually measured.

A task is one test case with defined inputs, relevant environment, and success criteria.
A trial is one attempt at a task under a recorded configuration.
An evaluator is the rule, program, person, or model that judges one aspect of a trial. Many tools call this a grader.
A trace records the material interactions during a trial: model responses, tool calls, results, state changes, and approvals.
An outcome is the final task-relevant state or effect.
An evaluation suite is a maintained collection of tasks designed to measure named capabilities, risks, and regressions.

Anthropic’s agent-evaluation article uses similar distinctions and emphasizes that an agent evaluation measures the model and its harness together (Anthropic, 2026). These definitions are useful, but they are not a universal standard; preserve source-specific terminology when integrating a tool.

Evaluate output, process, and outcome

A multi-step system can succeed or fail at several levels.

Output

Did the produced answer or artifact satisfy its requirements?

Checks may cover correctness, relevance, completeness, grounding, policy, structure, and communication. Exact match is appropriate for a known identifier or classification. It is usually too narrow for a useful explanation or design proposal with many valid forms.

Process

Did the system use an admissible path?

Material process criteria may include permitted data access, correct tool arguments, required approval, budget limits, provenance, no modification of protected resources, and no duplicate side effect. Do not require one canonical reasoning trace when several strategies are valid. Evaluate process only where the route itself carries risk, obligation, or diagnostic value.

Outcome

Did the intended change actually occur?

An agent can say “the reservation is complete” while no reservation exists. A patch can read well while tests fail. A support reply can be accurate while the customer’s problem remains unresolved. Outcome checks inspect the environment or later signal rather than trusting the system’s statement about success.

These levels can disagree. A correct outcome reached through unauthorized data access is not acceptable. A compliant process ending in the wrong result is not successful. A polished output with neither verified process nor outcome is evidence only about presentation.

Build representative tasks

An evaluation suite is a model of expected work. Its usefulness depends on what it includes and excludes.

Start with four categories:

Typical cases: frequent work the system is explicitly designed to support.
Edge cases: rare but legitimate inputs, ambiguous requests, missing data, unusual formats, long context, and tool errors.
Adversarial cases: attempts to bypass policy, redirect tools, expose data, or exploit trust boundaries.
Consequential cases: actions where a plausible mistake has high cost, weak reversibility, or a legal or ethical effect.

Include important slices rather than relying only on an overall average: languages, user groups, task types, data sources, tool paths, consequence levels, and environmental conditions. A high aggregate score can conceal failure in a small but important group.

Task sources can include product requirements, domain-expert cases, historical incidents, production traces that are lawfully and safely retained, user corrections, and deliberately constructed stress tests. Synthetic cases can broaden coverage but should not be assumed to reproduce real use.

Separate a development set used for iteration from held-out evidence used to estimate generalization. Repeated optimization against a fixed suite can make that suite less informative. Refresh it with newly discovered cases while preserving stable regression anchors.

Match the evaluator to the claim

No evaluator type is best for every criterion.

Evaluator	Good fit	Important limitation
Deterministic rule or schema	Format, ranges, authorization, exact fields, invariants	Cannot judge semantic quality beyond encoded rules
Executable check	Code, queries, simulations, environment state, side effects	The test environment may omit real conditions or encode incomplete requirements
Reference comparison	Classification, extraction, bounded questions with known answers	References can be incomplete and may punish valid alternatives
Human review	Nuance, domain correctness, consequence, evolving requirements	Expensive, slow, inconsistent, and subject to bias or fatigue
Model-based evaluation	Scalable rubric-based comparison or classification	Inherits model bias, can be manipulated, and is not ground truth
User or business outcome	Real usefulness and external effect	Often delayed, confounded, sparse, or shaped by who chooses to use the system

Combine evaluators when the claim has several parts. A research response might need deterministic citation-format checks, source verification, a domain rubric for synthesis, and a separate test for appropriate abstention.

OpenAI’s evaluation documentation recommends task-specific evaluations, representative production and expert data, continuous evaluation, and calibration of automated scoring with human feedback. It also notes position and verbosity biases in model judges (OpenAI, “Evaluation best practices”). Because this is vendor product material rather than an independent standard, validate the advice and any evaluator on the local task.

Treat model evaluators as instruments

A model-based evaluator can make semantic evaluation affordable at scale, but its score is a measurement produced by another variable model. Before using it for release decisions:

write a specific rubric with examples of material distinctions;
create a human-reviewed calibration set;
measure agreement and inspect disagreement, not only the average score;
test order, length, style, identity, and prompt-injection sensitivity;
use blinded or pairwise comparison when it better matches the question;
version the evaluator model, prompt, rubric, and reference material; and
retain a path for human review when the evaluator is uncertain or consequences are high.

High agreement on one dataset does not establish validity after the task distribution, system output style, or evaluator model changes. Recalibrate when the instrument changes.

Variable behavior requires trials, not anecdotes

One task run is a sample, not a reliability estimate. When behavior can vary, run enough trials to answer the product question and record model, prompt, context, tools, policy, and environment versions.

The preceding uncertainty chapter explains pass@k, pass^k, and why candidate generation differs from repeated effectful execution. The evaluation principle here is simpler:

use multiple trials when repeated use or sampling is part of the product;
aggregate results in a way that matches that use;
separate run-to-run variation from deliberate changes in configuration or environment; and
report counts and failure patterns, not only a polished average.

Ten sampled drafts with a reliable verifier can be a valid design. Ten purchase attempts are not a reliability method if any attempt may commit. The evaluation configuration must represent the actual control flow.

Evaluation continues after release

Offline tasks are necessarily incomplete. Production introduces new users, new inputs, changing data, tool outages, policy changes, and behavior not anticipated by the suite.

Use production evidence to discover—not automatically define—new evaluation cases:

explicit user corrections and appeals;
failed or abandoned tasks;
human overrides and escalations;
unexpected tool sequences or repeated retries;
policy blocks and security events;
outcome reversals or later reconciliation failures; and
distribution shifts in inputs, sources, cost, or latency.

Production telemetry must follow privacy, consent, access, and retention requirements. A useful trace is not permission to store every prompt, document, or hidden rationale indefinitely.

Earlier production-ML work made the broader point that model quality lives inside a system of data, code, infrastructure, and monitoring. The ML Test Score therefore evaluates readiness beyond offline model metrics (Breck et al., 2017). Generative systems need the same system view, extended to open-ended output and multi-step action.

Trade-offs

Choice	Benefit	Cost or risk
Narrow, executable criteria	Fast and repeatable regression checks	Can optimize what is easy to measure instead of what matters
Broad human rubrics	Capture nuance and evolving judgment	Expensive, inconsistent, and difficult to run continuously
Large suites	Wider coverage and better rare-case detection	Maintenance cost, slower iteration, and more false confidence if cases are weak
Stable suites	Comparable results over time	Overfitting and declining resemblance to production
Frequently refreshed suites	Better coverage of emerging failures	Harder longitudinal comparison and risk of contaminated labels
Trace evaluation	Finds unsafe routes and supports diagnosis	Can reject valid alternative paths and increase data sensitivity
Outcome evaluation	Measures the effect that matters	Outcomes can be delayed, noisy, or affected by factors outside the system

The goal is not maximum evaluation volume. It is enough evidence to make the next decision responsibly.

Failure modes

Vibe-based evaluation

A small set of memorable demonstrations substitutes for explicit criteria and representative cases. Improvements and regressions become matters of persuasion.

Benchmark substitution

A public model score is treated as proof that the application works, even though its context, tools, policies, and users were never tested.

One score hides several failures

Correctness, safety, cost, latency, user effort, and recovery are averaged into a number that no longer answers a clear question.

Golden-answer overreach

One reference response becomes the only acceptable answer for an open-ended task. Useful variation is penalized while unsupported responses with similar wording can pass.

Evaluator circularity

The same model family produces and judges an answer under similar missing evidence, so agreement is mistaken for independent verification.

Trajectory imitation

The suite rewards one expected tool sequence even when another safe route reaches the required outcome. The system learns to resemble the test rather than solve the task.

Test-suite comfort

Scores improve on a static suite while production failures change. No mechanism adds incidents, corrections, or distribution shifts back into evaluation.

Observability without evaluation

The system stores extensive traces but has no criteria for interpreting them. Data volume grows without producing a release decision or engineering lesson.

Fieldbook guidance

Write a minimum evaluation brief before increasing model discretion or environmental reach:

Capability and intended users:
Task outcome:
Supported and unsupported cases:
Acceptable variation:
Unacceptable output, process, and outcome:
Typical, edge, adversarial, and consequential tasks:
Configuration recorded for each trial:
Number of trials and aggregation rule:
Deterministic evaluators:
Human evaluators and rubric:
Model evaluators and calibration evidence:
Outcome evidence:
Important slices:
Release or escalation decision supported:
Production signals that may add new cases:

If the team cannot complete this record, it may still run an experiment. It does not yet have evidence for a dependable operating claim.

Checklist

References and how they are used

Anthropic, “Demystifying evals for AI agents” (2026) — Anthropic engineering article supporting the distinctions among tasks, trials, graders, traces, outcomes, harnesses, and suites, as well as evaluating a model and agent harness together. It is not a statistical standard.
OpenAI, “Evaluation best practices” — OpenAI-maintained API documentation for OpenAI evaluation workflows, used for task-specific and continuous evaluation, representative datasets, human calibration, and known model-judge biases. Its examples and product advice require local validation and are not an independent evaluation standard.
Bommasani et al., “Holistic Evaluation of Language Models” (2023) — peer-reviewed model-evaluation research supporting multiple scenarios, metrics, trade-offs, and transparent artifacts. It evaluates models rather than complete deployed products.
Breck et al., “The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction” (2017) — production-ML research supporting testing and monitoring across the wider system. It predates current generative models and agent trajectories.
NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1 (2024) — voluntary lifecycle risk-management profile supporting use-case-appropriate validation, multiple evaluation methods, documentation, feedback, and oversight.
Google, “Production ML systems: Monitoring pipelines” and “Dividing the original dataset” — Google educational material supporting important data slices, real-world metrics, held-out data, and the risk that repeatedly used test sets become less informative.