Evaluation Systems and Quality Gates

The Foundations chapter defined evaluation as the discipline of turning intended behavior into evidence. Production engineering asks a harder question: can the organization produce comparable evidence repeatedly while the model, prompts, context, tools, policies, data, evaluators, and environment all change?

A notebook that once showed 84% accuracy is not an evaluation system. Without the exact tasks, configuration, harness, environment, evaluator versions, trial records, and decision rule, the number cannot reliably explain whether a proposed change is better, worse, or merely different.

Thesis

An evaluation becomes a production control only when the claim, tasks, environment, harness, system configuration, evaluators, trials, and decision rule are maintained as one versioned measurement system.

A quality gate consumes that evidence to block, escalate, or permit a specific transition. Passing it supports only the decision and operating claim it was designed for. It does not certify that the system is universally correct, safe, or ready for every future condition.

The evaluation system is larger than the suite

An evaluation suite is one part of the production system. The full measurement path includes:

product claim and risk boundary
  -> coverage map and tasks
  -> environment fixtures and harness
  -> system configuration under test
  -> trials and captured evidence
  -> evaluators and adjudication
  -> estimates, uncertainty, and validity review
  -> quality-gate decision
  -> diagnosis, change, and suite maintenance

If any part changes, score comparability may change. OpenAI’s current third-party agent evaluation article emphasizes that model name alone is insufficient: prompts, tools, control logic, safeguards, retries, budgets, and environment help define the tested system and the behavior that can be elicited (OpenAI, 2026). The same requirement applies to internal product evaluation even when the purpose is regression detection rather than public comparison.

Version the measurement system

Treat the following as production assets rather than disposable test inputs:

Asset	What must be recoverable
Claim and coverage map	Intended behavior, excluded behavior, risk classes, user and task slices
Task	Input, initial state, allowed resources, success criteria, provenance, review status
Dataset or case collection	Membership, sampling rule, labels, splits, retention and access constraints
Environment fixture	Services, data snapshot, time, permissions, seeds, failure injection, cleanup behavior
Harness	Instructions, context assembly, interfaces, control flow, budgets, retries, stopping rules
System configuration	Models, providers, parameters, prompts, retrieval, policies, code, dependencies, feature flags
Evaluator	Program, model, prompt, rubric, references, calibration set, fallback and adjudication rules
Run record	Asset versions, trial identifiers, timestamps, cost, latency, traces, outcomes, errors
Report and decision	Estimates, uncertainty, slices, validity findings, gate result, owner, exception and expiry

Content hashes or immutable artifact identifiers are preferable to labels such as “latest prompt” or “production model.” Secrets and private data should not be copied into reports, but the record must identify the protected version or snapshot used.

Reproducibility does not mean every variable output repeats exactly. It means the team can reconstruct the conditions, rerun an equivalent experiment, distinguish intended randomness from uncontrolled change, and explain material differences.

The UK AI Security Institute’s open-source Inspect framework is one concrete example of evaluations represented as composable code: tasks, solvers, tools, scoring, logs, and extensions can be versioned together (Inspect AI, pinned 2025-11-28 tag). The architectural lesson does not require adopting Inspect or any other platform. A small team can begin with a repository, structured result files, and reproducible scripts if those assets preserve the same information.

Organize suites around decisions

One large “agent score” rarely answers a production question. Build a coverage map that links each operating claim to tasks, important slices, evaluators, and decisions.

Useful suite dimensions include:

product capability, such as research synthesis, refund handling, or patch generation;
consequence, reversibility, and required authority;
typical, edge, adversarial, consequential, and previously failed cases;
user group, language, geography, data source, tenant, and accessibility need;
component or path, such as retrieval, routing, approval, execution, and recovery;
environment condition, including stale data, dependency failure, long context, and interrupted work; and
metric family, including outcome quality, policy compliance, reliability, latency, cost, and user effort.

ISO/IEC 25059:2023 supplies an international quality model for specifying and evaluating several AI-system quality characteristics (ISO/IEC 25059:2023). It can help check whether a requirement set is too narrow, but it does not define this Fieldbook’s suite structure or an agent release gate. The published standard is also under revision; record the edition actually used.

Keep stable regression anchors so that results remain comparable over time, but maintain a governed path for adding production failures, expert cases, new threats, and changed product requirements. Separate at least:

a development set that engineers may inspect while iterating;
held-out cases used for a less biased decision estimate;
restricted adversarial or high-consequence cases whose leakage would weaken the test; and
retired or quarantined cases retained with the reason they no longer support the claim.

Do not silently delete a difficult case because the system cannot pass it. Change or retire a case only through review of the requirement, ground truth, environment, or validity.

Make the environment part of the test

Agent behavior depends on the world it can observe and change. An evaluation environment should define:

initial authoritative state and reset behavior;
identities, permissions, credentials, and approval rules;
available interfaces and their versions;
network, clock, locale, and dependency conditions;
token, turn, time, retry, concurrency, and monetary budgets;
whether browsing or retrieval can expose answers;
how committed effects are isolated and verified; and
which logs and artifacts are retained.

Use isolated tenants, synthetic accounts, sandboxes, service doubles, or replayable snapshots when real operations would create unacceptable effects. A simulation must preserve the semantics that matter to the claim. A fake payment API that always returns success cannot test reconciliation; a coding task whose repository history contains the answer cannot test problem solving.

There is a tension between standardization and elicitation. A fixed harness helps attribute differences across configurations. A model-specific or stronger harness may reveal capability or failure that a common interface suppresses. State which question the run answers:

comparative regression: keep the task, harness, budgets, and evaluators fixed enough to compare a proposed configuration with its baseline;
capability discovery: use a strong credible harness and sufficient budget to find what the system can do;
safety or abuse testing: model a credible adversary and record the attack resources rather than testing only a convenient prompt.

Do not mix results from these purposes into one table without qualification.

Treat evaluators as measurement instruments

An evaluator can regress even when the system under test does not. Code can contain a bug, a reference can become stale, human reviewers can drift, and a judge model update can change the scale.

For every material evaluator:

define the claim and failure distinctions it judges;
test it against human-reviewed or otherwise verified cases;
include positive, negative, borderline, adversarial, and invalid inputs;
measure disagreement and inspect its pattern;
record false accepts and false rejects where ground truth permits;
test sensitivity to output order, verbosity, style, identity, and prompt injection when relevant;
version code, model, prompt, rubric, references, and thresholds; and
define recalibration and human-adjudication triggers.

Agreement is not validity. Two judges may agree because they share a blind spot. Human consensus may also be a weak reference when reviewers lack the required domain knowledge or see incomplete evidence. Name who is qualified to adjudicate which criterion.

Anthropic recommends combining grader types, calibrating model-based rubrics against expert judgment, reading transcripts, and using production monitoring and human review alongside automated evaluation (Anthropic, 2026). This is useful provider-published production material, not a guarantee that one grader mix works for every application.

Report variation and uncertainty

When behavior varies, a point estimate such as 91% passed is incomplete. At minimum report:

the numerator, denominator, number of tasks, and trials per task;
the unit being averaged: trial, task, user, conversation, or another cluster;
the sampling and aggregation rule;
the estimate and an appropriate uncertainty interval;
paired differences when the same tasks compare two configurations;
important slice results and missing data;
invalid, timed-out, refused, or indeterminate trials; and
the practical effect size relevant to the decision.

For a simple binary rate from independent trials, a binomial interval such as a Wilson interval is generally more informative than an unqualified percentage, especially near zero or one or with small samples (Wilson, 1927). Agent evaluations often violate the independent-trial assumption: repeated attempts on one task are clustered, the same environment serves several cases, evaluators are noisy, and adaptive retries change exposure. Use paired, hierarchical, bootstrap, or other methods only when their assumptions match the design and are documented.

Do not adopt a universal trial count or p < 0.05 rule. Before running an expensive gate, decide:

what degradation or improvement would matter to the product;
how much uncertainty is tolerable for this consequence level;
whether the gate tests an absolute requirement, a comparison, or both;
which slices need their own minimum evidence; and
what happens when the suite cannot resolve the decision.

Statistical significance does not make a difference important. Lack of significance does not prove equivalence. A single confirmed unauthorized transfer may block a release even when the overall pass rate is high, while a tiny statistically clear style change may have no product consequence.

Design gates as decision records

A useful quality gate has several independent parts:

transition being governed
system change and candidate configuration
baseline and comparison method
absolute requirements
non-regression limits
target improvements
required slices and minimum evidence
validity and evaluator-health checks
cost and latency bounds
decision owner and escalation path
exception authority, compensating controls, and expiry
result, evidence links, rationale, and timestamp

Separate three types of rule:

Hard requirements: confirmed policy, security, privacy, authorization, data-integrity, or other unacceptable failures cannot be averaged away.
Non-regression rules: important metrics and slices may not degrade beyond a defined practical margin, with uncertainty considered.
Targets: desired improvements guide optimization but may not block every transition.

Avoid collapsing them into a weighted score. A large latency improvement must not compensate for unauthorized data disclosure. A higher average answer-quality score must not conceal a critical-language regression. If a trade-off is legitimate, make the person with the relevant product or risk authority decide it explicitly.

A gate has more than pass and fail. Useful outcomes are:

pass: required evidence supports the transition;
fail: a defined blocking condition is met;
inconclusive: evidence is too noisy, incomplete, invalid, or inconsistent;
escalate: the trade-off or consequence requires a named decision owner; and
exception granted: an authorized owner accepts a time-bounded deviation with rationale, compensating controls, and follow-up work.

Changing a threshold after seeing an inconvenient result is a policy change, not a test fix. Preserve the old decision, explain the new reasoning, and re-evaluate affected evidence.

Use a tiered evaluation cadence

Not every suite belongs in every development loop. A practical cadence is:

Stage	Purpose	Typical evidence
Local or pull-request smoke	Catch obvious contract and behavior regressions quickly	Small deterministic subset, evaluator tests, fixed seeds where useful
Pre-merge or continuous integration	Protect common capabilities and hard invariants	Stable regression suite, paired candidate–baseline runs
Scheduled broad run	Detect slice, variability, and dependency problems	Larger suites, multiple trials, adversarial and failure conditions
Pre-release qualification	Support a named promotion or capability-expansion decision	Held-out and restricted cases, full configuration record, validity review, gate sign-off
Shadow or canary evidence	Check production-like traffic and infrastructure before broad exposure	Controlled traffic, outcome reconciliation, rollback signals, user-impact limits
Production feedback	Discover drift and unanticipated failure	Outcome metrics, incidents, corrections, appeals, sampled review

Anthropic reports automated evaluations as useful in development and CI, with production monitoring, feedback, transcript review, A/B testing, and human studies covering different gaps (Anthropic, 2026). Earlier production-ML work likewise argues that readiness depends on tests and monitoring across data, model, infrastructure, and serving behavior, not offline model quality alone (Breck et al., 2017).

The Testing, Change, and Release chapter will own the mechanics of CI, rollout, feature flags, compatibility, and rollback. This chapter owns the behavioral evidence and decision semantics those mechanisms consume.

Audit validity, not only scores

The evaluation system itself needs quality assurance. Review samples from successes, failures, and borderline cases. Investigate:

broken, ambiguous, impossible, duplicated, or mislabeled tasks;
environment leakage, flaky dependencies, stale snapshots, and cleanup failures;
contamination through training data, public answers, browsing, repository history, or memory;
reward hacking and unintended shortcuts through the scorer or harness;
refusals or safeguards that alter what the score means;
evaluation awareness or behavior unlike normal deployment;
evaluator drift, prompt injection, or systematic preference for style over substance;
suite overfitting caused by repeated developer exposure; and
mismatch between the reported aggregate and important production slices.

OpenAI’s 2026 audit of coding evaluations found enough broken tasks in a prominent benchmark to change the interpretation of its scores, illustrating that a benchmark can lose validity even while its execution remains reproducible (OpenAI, 2026). The reported defect rate is specific to that audit; the durable lesson is to test the tests.

Record invalid trials separately rather than counting every infrastructure failure as model failure or silently dropping it. If exclusions materially change the result, report both the rule and its effect.

Operate the feedback loop

An evaluation system needs owners and maintenance work:

a capability owner defines the operating claim and supported task distribution;
domain experts own ground truth and semantic rubrics;
risk or policy owners define hard requirements;
evaluation maintainers own harnesses, environments, calibration, and reports;
service owners connect production outcomes and incidents back to cases; and
a named decision owner accepts or rejects the transition.

Track suite health separately from system performance: task age, invalid-case rate, flaky-run rate, evaluator disagreement, unreviewed production failures, runtime, cost, coverage gaps, exception count, and time since calibration.

Adding every production trace is neither safe nor useful. Apply privacy, consent, access, retention, and de-identification rules; select cases because they represent a claim, failure, or distribution shift; have qualified reviewers establish expected behavior; and preserve provenance back to the governed incident or feedback source.

Trade-offs

Choice	Benefit	Cost or risk
Fixed suite and harness	Strong longitudinal comparison	Overfitting and declining production relevance
Frequently refreshed cases	Captures new behavior and failures	Weakens direct comparison and increases label maintenance
Large repeated runs	Narrower uncertainty and better slice visibility	Cost, latency, carbon use, and slower iteration
Strict blocking gates	Prevents known unacceptable changes	False blocks, gaming, and slower learning if criteria are poor
Flexible expert decisions	Handles context and legitimate trade-offs	Inconsistency and weak accountability without records
Model-based scoring	Scalable semantic judgment	Bias, drift, manipulation, and circularity
Production experiments	Stronger realism	User exposure, confounding, delayed outcomes, and operational risk
One shared platform	Consistent records and easier governance	Central bottleneck and poor fit for unusual domains

The goal is not maximum automation. It is a measurement and decision process whose uncertainty, cost, and failure modes are appropriate to the consequences it governs.

Failure modes

The score without the tested system

A report names only the model and percentage. Prompt, tools, harness, safeguards, budgets, environment, and evaluator are missing, so the result cannot support comparison or reproduction.

One aggregate gate

Quality, policy, security, latency, and cost are weighted into one number. Severe failure in a small or non-compensable category disappears inside the average.

Point-estimate theater

Two configurations score 82% and 84%, and the second is declared better without sample counts, paired analysis, uncertainty, or practical importance.

Green suite, broken tasks

The system optimizes against invalid ground truth, leaked answers, flaky environments, or exploitable scoring. Nobody reviews the evaluation assets because only candidate failures receive attention.

Evaluator upgrade without recalibration

A judge model, rubric, or reference set changes and historical scores are compared as if the scale remained fixed.

Threshold editing after the result

A failed candidate causes the team to lower a gate silently. The decision is hidden inside test configuration instead of owned as a risk exception or policy change.

Production feedback warehouse

Every prompt and trace is retained “for evals,” creating privacy and security exposure while few cases are reviewed or converted into useful tasks.

Permanent exception

A temporary bypass has no owner, expiry, compensating control, or required follow-up, so the gate becomes optional for influential teams.

Fieldbook guidance

For every material gate, maintain a record like:

Decision and transition:
Operating claim:
Candidate and baseline configuration IDs:
Suite, environment, harness, and evaluator versions:
Tasks, trials, slices, and sampling rule:
Absolute blocking requirements:
Non-regression limits and practical margins:
Improvement targets:
Statistical method and assumptions:
Invalid or excluded trials:
Validity review findings:
Quality, risk, cost, and latency results:
Gate outcome:
Decision owner and timestamp:
Exception, compensating controls, and expiry:
Evidence links and follow-up cases:

If the team cannot reconstruct this record, it has a score but not yet a dependable quality gate.

Checklist

References and how they are used

Anthropic, “Demystifying evals for AI agents” — Anthropic engineering article supporting realistic tasks, combined evaluators, transcript review, CI use, production monitoring, and ongoing suite maintenance. Its recommendations come from Anthropic and selected customer experience rather than an independent standard.
OpenAI, “A shared playbook for trustworthy third party evaluations” — OpenAI research and engineering article supporting complete tested-system records, harness and budget disclosure, validity checks, and explicit claim boundaries. It focuses on frontier and third-party evaluations, so this chapter adapts rather than copies its scope.
OpenAI, “Separating signal from noise in coding evaluations” — current primary audit used as concrete evidence that tasks and scoring can invalidate a benchmark. Its defect estimates are not generalized beyond the coding datasets studied.
ISO/IEC 25059:2023, “Quality model for AI systems” — published international standard used to support multi-characteristic AI quality requirements and completeness review. The public abstract does not provide an agent gate design, and a revision is underway.
ETSI TR 103 910 V1.1.1, “AI Testing” — published European technical report supporting explicit AI test methodology and specification. It is not a harmonized standard and is broader than generative agents.
Breck et al., “The ML Test Score” — production-ML research supporting tests and monitoring across system components, canarying, and readiness beyond offline model scores. The exact rubric predates generative agents.
UK AI Security Institute, Inspect AI — canonical open-source repository at tagged commit b547763 (release/2025-11-28), used as an implementation example of versioned, composable evaluation assets. It is not presented as the preferred platform.
Wilson, “Probable Inference, the Law of Succession, and Statistical Inference” — foundational statistical source supporting interval estimates for simple binomial proportions. Agent trials often require richer designs, so the chapter does not prescribe Wilson intervals universally.