Observability and Incident Response

An agentic system can return a valid HTTP response while failing the user’s task. It can call every interface successfully yet change the wrong record, disclose the wrong data, loop without progress, or stop before an external effect is confirmed. Conventional service telemetry remains necessary, but it does not by itself explain whether the system exercised the right authority, used trustworthy context, or produced the intended result.

This chapter connects production evidence to operational response. It does not redefine the run state machine, evaluation methodology, retry semantics, security controls, or release process covered elsewhere. Its question is narrower:

When a production system behaves unexpectedly, can the responsible team determine what users experienced, reconstruct the material path and effects, contain further harm, and learn enough to prevent or detect recurrence?

Thesis

An agentic system is observable only when its telemetry, state records, and outcome evidence can explain task-relevant behavior and support a bounded operational decision. Incident response must be designed with the system’s authority and control points, not improvised after harm occurs.

Collecting more data is not the same as becoming more observable. A useful design begins with questions the team must answer and controls responders can actually operate. It then records the smallest evidence set that answers those questions within privacy, security, cost, and retention constraints.

From telemetry to an evidence model

Production evidence has several complementary forms:

Evidence	Best at answering	What it cannot establish alone
Metrics	How often, how much, and whether an aggregate is changing	Why one run behaved as it did or whether a specific effect occurred
Events and logs	Which discrete fact, decision, error, or transition was recorded	The complete causal path unless records are correlated
Traces and spans	Which operations formed one path, their parent–child relationships, timing, and status	Whether a fluent output or technically successful call satisfied the task
Authoritative state and effect records	What the system of record accepts as true and which external changes were committed	Why the system chose the path unless joined to decision records
Outcome evidence	Whether named success, policy, and user-impact conditions were met	Every internal cause of a failure
Sampled artifacts	The semantic detail needed for expert diagnosis, such as selected inputs, outputs, or retrieved passages	Fleet-wide frequency; they may contain sensitive data and sampling bias

OpenTelemetry standardizes common names and attributes across traces, metrics, logs, and events, while W3C Trace Context defines interoperable HTTP headers for propagating trace identity across services (OpenTelemetry Semantic Conventions; W3C Trace Context). These standards solve important transport and correlation problems. They do not define product success, authority, or incident severity for an agentic application.

Use each evidence type for its purpose. Do not turn a trace into a transcript warehouse or a metric label set into an unbounded database. High-cardinality identifiers belong in logs or traces rather than fleet-wide metric dimensions unless the monitoring backend and query need justify the cost.

Establish one correlation spine

A responder should be able to move from a user report or alert to the relevant run, operations, effects, configuration, and outcome without searching unrelated stores by timestamp and guesswork.

At minimum, relate the identifiers appropriate to the product:

user-visible request or task
  -> conversation or case
  -> run and trace
  -> step, span, model call, interface operation, approval, and child run
  -> external effect or artifact
  -> verifier result, user correction, incident, and evaluation case

Not every system needs every identifier, and one identifier should not be forced to mean all of them. A conversation may contain several runs; one run may produce several external effects; an asynchronous effect may outlive its initiating trace. Preserve explicit relationships.

OpenAI’s Agents SDK, for example, represents an end-to-end workflow as a trace containing model, function, guardrail, and handoff spans, and provides a group identifier for linking related traces (OpenAI Agents SDK tracing). Anthropic’s Claude Code telemetry uses a prompt identifier to join the API and tool events caused by one prompt and a tool-use identifier to join decisions, execution, and results (Anthropic Claude Code monitoring). These are useful implementation examples, not universal field names.

Correlation identifiers should be opaque, collision-resistant, and scoped to their purpose. Do not put email addresses, raw prompts, tenant secrets, or other personal data into trace identifiers. Treat incoming trace context as untrusted metadata: it may be malformed or chosen by an attacker, and it must not grant access to another tenant’s evidence.

Record decisions and effects, not private reasoning

The record should explain what happened without claiming access to a model’s private chain-of-thought. For each material transition, retain enough structured data to answer:

• Which goal, task state, and configuration were active?
• Which model, prompt or instruction version, context manifest, policy, and routing decision applied?
• What operation or lifecycle transition was proposed?
• Which schema, permission, approval, budget, and policy checks ran, and what did they decide?
• What was attempted, by which principal, against which resource and state version?
• What result, error class, latency, token use, cost, retry count, and provider request identifier were observed?
• Was the external effect confirmed, rejected, partial, or unknown?
• Which verifier accepted or rejected progress and completion?
• What user correction, appeal, or downstream signal later changed the interpretation?

The exact model prompt and full interface arguments are not automatically required. Often a content hash, protected artifact reference, redacted field set, or allowlisted excerpt provides enough reproducibility with less exposure. A generated rationale may help a user or reviewer, but it is not a faithful log of hidden computation and should not be treated as authoritative evidence.

The Architecture chapters own the meaning of authoritative state, execution boundaries, approvals, and run transitions. Observability owns preserving their identifiers and decisions together so production behavior can be reconstructed.

Observe the whole system, not only the model call

Model latency and token counts are useful but narrow. A production view should cover the path that determines outcomes:

1. Demand and task: entry point, supported task class, user or tenant slice, consequence level, and declared success condition.
2. Configuration: application, model, provider, prompt, context policy, routing, interface, evaluator, feature-flag, and deployment versions.
3. Context assembly: sources selected, provenance and trust labels, age, exclusions, compaction, size, and access decisions—without copying all source content by default.
4. Control loop: state transitions, decisions, validation, progress signals, budgets, repeated steps, waiting, cancellation, and terminal reason.
5. Capabilities and authority: operation requested, principal, permission and approval decision, policy version, execution-boundary result, and committed effect.
6. Dependencies: model and interface requests, queues, retrieval, storage, networks, rate limits, timeouts, retries, and fallback paths.
7. Outcome: verifier evidence, external state reconciliation, user-visible result, correction, abandonment, escalation, and later downstream consequence.

This yields an important distinction:

Protocol success is evidence about an operation. Task success is evidence about the intended outcome.

An HTTP 200, a model end_turn, or a Tool result marked successful may be necessary for a path, but none is a general success criterion. Conversely, a model refusal or policy block is not automatically an incident; it may be the system working correctly. Alert on user-impacting symptoms and violated operating claims, then use internal signals to diagnose them.

Choose production signals from operating claims

Start with what the system promises and what can cause unacceptable consequences. Useful signal families include:

• verified task completion, incomplete and indeterminate outcomes, user corrections, abandonment, and escalation;
• confirmed, rejected, partial, duplicate, and unknown external effects;
• policy denials, approval requests, approval bypass attempts, and operations outside the expected authority envelope;
• no-progress cycles, repeated operations, budget exhaustion, recursion depth, and cancellation latency;
• model, retrieval, interface, queue, and state-store errors, saturation, latency, and fallback use;
• cost, tokens, steps, tool use, and elapsed time per completed or failed task;
• configuration, model, provider, route, context-source, user, language, geography, tenant, and consequence slices; and
• telemetry health itself: export failures, dropped spans, sampling changes, clock skew, schema violations, and missing correlations.

Rates need meaningful denominators. “Thirty unknown effects” is different across forty and forty million attempts. “Average cost per request” can improve merely because more tasks fail early; cost per verified outcome and the joint quality–cost distribution may reveal the real change.

Do not label every uncertain semantic signal as fact. A model-based production classifier should record its model, prompt, rubric, threshold, calibration evidence, and abstention behavior. Use it to prioritize or aggregate only within its measured error bounds, with stronger confirmation for consequential actions.

Detection needs several layers

No single detector covers variable, open-ended behavior. Combine layers with different strengths:

1. Deterministic invariants catch schema violations, illegal state transitions, missing approvals, duplicate effect identifiers, budget violations, and known unsafe patterns.
2. Outcome and service-level signals detect user-visible failure, excessive latency, abandoned tasks, or breached reliability objectives.
3. Baseline and slice monitoring detects material shifts by model, route, language, tenant, task, or version rather than hiding them in a fleet average.
4. Evaluation-derived monitors reuse validated criteria on production samples where the deployment population and privacy policy permit.
5. Expert sampling finds semantic failures that automated checks do not yet express.
6. User and partner reports surface delayed, contextual, or downstream effects unavailable to the runtime.

Google SRE incident-management material recommends timely, actionable alerts centered on user-facing symptoms, while retaining preventive alerts for conditions such as an imminent hard quota (Google SRE Incident Management Guide). For agentic systems, that means “verified completion fell below the objective” is usually a better page than “the model used five tools.” The latter may still be valuable as diagnostic context or a preventive signal when it predicts runaway cost.

Every paging alert should name an owner, user impact or risk condition, severity rule, initial evidence, and a tested response procedure. A dashboard without a decision or owner is situational awareness, not incident response. An alert that responders routinely ignore is a monitoring defect.

Define incidents by impact and required coordination

An anomaly becomes an incident when its actual or credible potential impact requires coordinated response beyond ordinary request handling. Relevant classes include:

• materially incorrect, harmful, discriminatory, or misleading outcomes;
• unauthorized, unintended, duplicate, irreversible, or unknown external effects;
• privacy, confidentiality, data-residency, or retention violations;
• corrupted authoritative state, memory, context sources, or evaluation records;
• runaway loops, excessive spend, capacity exhaustion, or dependency cascades;
• unexpected behavior after a model, provider, policy, prompt, interface, or data change;
• security compromise, prompt injection with material effect, credential misuse, or telemetry tampering; and
• monitoring or evaluator failure that removes confidence in safe operation.

Severity should reflect user harm, affected scope, continued spread, duration, reversibility, authority exercised, data sensitivity, legal obligations, and uncertainty about external effects. Do not derive it from model confidence, the number of exceptions in logs, or whether the root cause sounds “AI-specific.” One confirmed unauthorized transfer can be more severe than thousands of harmless refusals.

Security incidents may trigger specialized legal, forensic, and notification duties. NIST SP 800-61 Rev. 3 integrates incident response across preparation, detection, response, recovery, and broader cybersecurity risk management (NIST, 2025). This chapter applies the lifecycle more broadly to production agentic failures; it does not replace jurisdiction-specific security, privacy, safety, employment, or sector requirements.

Prepare control levers before the incident

Responders cannot contain a system through a dashboard. Design, authorize, document, and exercise controls such as:

• stop new runs while allowing evidence export and safe cleanup;
• disable one consequential capability or switch it to propose-only or read-only mode;
• isolate a tenant, user cohort, task class, region, model, provider, route, or configuration;
• revoke or narrow credentials and approval grants;
• reduce step, token, cost, concurrency, and external-effect budgets;
• pin or revert configuration and route to a known bounded fallback;
• pause, cancel, or quarantine runs while preserving resumable state;
• quarantine suspect context sources, memory records, outputs, and evaluation cases; and
• reconcile or compensate already committed external effects through the owning system.

A broad kill switch may reduce immediate harm but also interrupt critical work, destroy in-flight evidence, or create inconsistent state. Prefer controls with useful scope and explicit transition semantics. Test them in exercises and staged environments; an emergency control first tried during an incident is only a hypothesis.

The Reliability chapter will own idempotency, checkpoints, compensation, and durable recovery mechanisms. Testing, Change, and Release will own rollout and rollback mechanics. This chapter owns identifying which control to invoke, preserving evidence during its use, and verifying that containment changed the actual impact.

Run a disciplined response

A practical response lifecycle is:

1. Triage and declare

Confirm the signal without waiting for complete certainty. Open an incident record, assign severity, name an incident commander or equivalent coordinator, and identify operations and communications owners. Record what is known, unknown, assumed, and changing.

2. Protect users and contain

Prioritize limiting further impact over proving the root cause. Stop or narrow risky operations, protect credentials and data, isolate affected configurations or populations, and establish a safe fallback. Verify containment through outcome and state evidence, not merely a successful configuration command.

3. Preserve evidence

Protect the timeline, correlated trace and event references, configuration versions, policy and approval decisions, model and provider identifiers, state snapshots, effect records, user reports, and responder actions. Restrict access and preserve chain of custody where legal or forensic use may matter. Do not “clean up” the only evidence before capture.

4. Scope and diagnose

Determine affected users, tasks, tenants, data, effects, versions, time range, and remaining exposure. Compare configuration changes and healthy slices, inspect state transitions and dependency behavior, and reproduce only in a safe environment. Keep trigger, contributing conditions, failed controls, and root causes distinct.

5. Recover and reconcile

Restore service gradually under stronger monitoring. Confirm authoritative state, repair or compensate external effects, reprocess safe work, communicate residual uncertainty, and verify the user outcome. “The agent is responding again” is not recovery if wrong records or obligations remain.

6. Communicate

Provide regular, audience-appropriate updates about impact, scope, mitigations, workarounds, and uncertainty. Coordinate internal, customer, partner, provider, regulator, and public communication through authorized owners. Preserve timestamps and decisions; avoid speculative root-cause claims during active response.

7. Learn and close the loop

Write a factual, blameless post-incident review that covers impact, detection, timeline, response, causes, contributing conditions, where controls succeeded or failed, and where the team got lucky. Assign concrete preventive, detective, mitigative, and process improvements to owners with priority, due date, and verification. Add representative failures to evaluation suites and production monitors without copying sensitive incident data indiscriminately.

Google’s SRE material separates coordination, communication, and control during response and emphasizes that postmortems create value only when concrete actions are tracked (Incident Management Guide; Postmortem Culture). “Blameless” should improve disclosure and systems learning; it does not erase decision ownership, professional duties, or accountability for knowingly violating policy.

Protect the observability system

Telemetry can be one of the most sensitive datasets in the product. Model inputs, retrieved documents, interface arguments, outputs, file paths, identifiers, and error messages may contain personal data, secrets, proprietary content, or attack payloads.

Apply:

• collection purpose and data minimization;
• allowlisting, redaction, tokenization, or protected artifact references before export;
• separate controls for content capture rather than assuming metadata consent covers content;
• tenant isolation, least-privilege access, encryption, access audit, and export restrictions;
• retention and deletion rules matched to diagnostic, contractual, and legal needs;
• sampling policies that retain rare consequential failures without collecting every ordinary interaction; and
• integrity monitoring for collectors, pipelines, schemas, timestamps, sampling decisions, and storage.

OpenAI documents that generation and function spans can contain sensitive inputs and outputs and provides controls to disable that capture. Anthropic makes its OpenTelemetry export opt-in and gates more detailed Tool attributes and content separately. Those controls demonstrate the risk, but an application owner still must decide whether collection and export are permitted for its users, jurisdictions, and data classes.

Telemetry is also fallible. Missing spans do not prove an operation did not occur. A sampled trace does not represent all traffic. Clock skew can distort a timeline. A compromised service can emit false success. Confirm consequential effects against authoritative systems and monitor the monitoring path itself.

Trade-offs

Choice	Benefit	Cost or risk
Full content capture	Fast semantic debugging	Privacy, security, retention, access, and prompt-injection exposure
Metadata and protected references	Smaller and safer telemetry	Diagnosis may require controlled retrieval from another store
Head-based sampling	Predictable collection cost	Misses rare failures not known at trace start
Tail or outcome-aware sampling	Retains slow, failed, or consequential traces	More buffering, infrastructure, policy complexity, and delayed decisions
One shared schema	Cross-team queries and tooling	Lowest-common-denominator semantics and central coordination cost
Domain-specific events	Precise task and effect meaning	Harder fleet-wide comparison and migration
Automatic containment	Faster response	False positives can interrupt work or create new state transitions
Human incident declaration	Context-sensitive judgment	Delayed response when ownership or signals are unclear
Long retention	Supports delayed discovery and trend analysis	Greater breach impact, compliance burden, and storage cost

The goal is not a perfect replay of every internal token. It is sufficient, trustworthy evidence and prepared control for the consequences the system can create.

Failure modes

The trace-as-transcript warehouse

Every prompt, document, and interface payload is retained indefinitely. The team creates a high-value sensitive dataset without a defined diagnostic need, access boundary, or deletion path.

Successful requests, failed tasks

Dashboards show low HTTP error rates while verified completion declines or external effects remain unknown. Operations measure the infrastructure path but not the product claim.

Orphan evidence

Model calls, operations, approvals, effects, and user reports use unrelated identifiers. Responders reconstruct incidents through timestamps and intuition, losing both time and confidence.

Fleet averages hide the affected slice

Overall quality appears stable while one language, model route, tenant, task class, or high-consequence capability regresses sharply.

A model judge as the sole alarm

An uncalibrated classifier turns ambiguous output into a definitive incident signal. Its own drift, prompt sensitivity, false negatives, and manipulation are not monitored.

Dashboard without response ownership

The anomaly is visible, but no one is responsible for declaring an incident, operating a containment control, or communicating impact.

Kill switch without state semantics

An emergency stop terminates workers while writes are in flight, loses checkpoints, or silently resumes queued operations later.

Evidence destroyed during mitigation

Responders redeploy, delete sessions, or overwrite configuration before preserving the timeline and versions needed to determine scope.

Service restored, effects unreconciled

Traffic recovers, but duplicate payments, wrong permissions, corrupted memory, or unmet user obligations remain. The incident is closed around uptime rather than outcome.

Postmortem without owned actions

The review produces a polished narrative but no concrete changes to controls, monitors, evaluations, runbooks, or responsibility.

Fieldbook guidance

Maintain two linked operational contracts.

The telemetry contract should state:

Operating claims and user-impact signals:
Correlation identifiers and propagation rules:
Required event and span types:
Configuration and policy versions:
Outcome and external-effect evidence:
Metric definitions, units, denominators, and slices:
Content-capture, redaction, access, sampling, and retention policy:
Telemetry health indicators:
Alert owner, severity rule, and response procedure:
Known blind spots and unavailable provider detail:

The incident record should state:

Incident identifier, commander, operations owner, and communications owner:
Start, detection, declaration, containment, recovery, and closure times:
Observed and potential user impact:
Affected tasks, users, tenants, data, effects, and configurations:
Known facts, assumptions, unknowns, and evidence links:
Severity and notification obligations:
Containment actions and verification:
External-effect reconciliation and residual risk:
Timeline and responder decisions:
Trigger, contributing conditions, root causes, and failed controls:
What worked and where the system was lucky:
Corrective actions, owners, priority, due dates, and verification:
New evaluation cases, monitors, runbook changes, and follow-up review:

If the team cannot link an alert to this evidence and operate a bounded control, it has monitoring but not yet a dependable response capability.

Checklist

• Are production signals derived from operating claims, user outcomes, authority, and unacceptable consequences?
• Can a user report be correlated to the relevant run, model and interface operations, approvals, state changes, effects, configuration, and verifier evidence?
• Are metrics, events, traces, state records, outcome evidence, and sampled artifacts assigned distinct purposes?
• Do event records distinguish proposed, authorized, attempted, confirmed, partial, rejected, and unknown effects?
• Are protocol success and task success measured separately?
• Are important model, route, task, language, tenant, consequence, and configuration slices visible?
• Are model-based monitors calibrated, versioned, able to abstain, and backed by stronger confirmation where consequences require it?
• Does every paging alert have a user-impact or risk condition, owner, severity rule, evidence link, and tested response procedure?
• Can responders stop or narrow capabilities, populations, credentials, configurations, and budgets without losing evidence or corrupting state?
• Are containment controls exercised and their actual effects verifiable?
• Does the incident process assign coordination, operations, and communications responsibilities?
• Are evidence preservation, external-effect reconciliation, staged recovery, and residual uncertainty part of closure?
• Do post-incident actions have owners, priorities, due dates, and verification criteria?
• Are incident-derived evaluation cases and monitors added under privacy and provenance controls?
• Are sensitive content, identifiers, access, retention, deletion, sampling, and tenant isolation governed explicitly?
• Are telemetry export failures, dropped or orphan spans, schema drift, clock skew, and sampling bias monitored?

References and how they are used

• OpenAI Agents SDK, “Tracing” — OpenAI-maintained Agents SDK documentation used for representing workflows, model generations, function calls, guardrails, and handoffs as correlated traces and spans. Its sensitive-data defaults and SDK-specific schema are examples, not a universal observability design.
• Anthropic Claude Code, “Monitoring” — Anthropic-maintained Claude Code documentation used for exporting metrics, events, and optional traces through OpenTelemetry; correlating prompts, API requests, and Tool activity; and gating detailed content. It describes Claude Code rather than every Anthropic API application.
• OpenTelemetry Semantic Conventions — community specification used for common trace, metric, log, and event vocabulary. Generative-AI and agent conventions are still evolving, so the chapter does not freeze its domain model to the current draft fields.
• W3C Trace Context — W3C Recommendation used for interoperable propagation of trace identity across HTTP services. It correlates distributed requests but does not define task, tenant, approval, or external-effect identity.
• Google SRE, “Incident Management Guide” — Google SRE incident-management material supporting actionable symptom-based alerts, prepared response, clear incident roles, communication, control, and learning. It reflects Google’s operating experience and must be scaled to local organizations.
• Google SRE Workbook, “Postmortem Culture: Learning from Failure” — Google SRE workbook chapter supporting factual timelines, measured impact, blameless learning, specific ownership, and tracked actions. Its organizational practices are examples rather than a formal standard.
• NIST SP 800-61 Rev. 3, “Incident Response Recommendations and Considerations for Cybersecurity Risk Management” — final 2025 US public-sector incident-response publication used to connect preparation, detection, response, recovery, and improvement with risk management. It governs cybersecurity incidents; this chapter adapts the lifecycle to broader agentic-system failures without claiming equivalent legal scope.