Reliability, Recovery, and Durable Execution

Long-running agentic work crosses unreliable boundaries: model providers, networks, queues, storage, user approvals, external APIs, worker processes, and changing deployments. Any one of them can fail after accepting work but before reporting its result. A process restart is therefore not the difficult case. The difficult case is recovering without losing accepted progress, repeating a consequential effect, trusting stale state, or declaring success while the world remains inconsistent.

This chapter assumes the explicit run states, typed outcomes, authoritative state, and execution contracts defined in Architecture. It also assumes the production evidence and response process from Observability and Incident Response. Its responsibility is the machinery that allows a run to wait, fail, restart, and continue safely.

Thesis

Reliable agentic execution does not require every model decision or dependency to be deterministic. It requires durable records of accepted progress, explicit effect semantics, retry rules tied to failure class, and recovery that revalidates the current world before continuing.

Durability is not “keep the agent process alive.” A durable run may spend most of its lifetime with no worker or model active. Its state, pending obligations, timers, and resume conditions survive elsewhere. When execution resumes, the system reconstructs what is known, resolves what is uncertain, and performs only work that is still authorized and necessary.

Define the reliability claim

“The agent is reliable” is too vague to engineer. State which user and system properties must survive failure:

• accepted requests are not silently lost;
• committed progress can be reconstructed within a stated loss tolerance;
• one logical intent does not create duplicate consequential effects;
• partial and unknown effects remain visible until reconciled;
• waits, approvals, deadlines, cancellation, and ownership survive process restarts;
• stale workers and concurrent resumes cannot overwrite newer truth;
• recovery completes within an appropriate time and does not overload dependencies; and
• the final result reflects reconciled external state, not merely a resumed conversation.

Measure the properties separately. Useful objectives include recovery time, maximum accepted-state loss, duplicate-effect rate, unknown-effect age, successful-resume rate, compensation backlog, stale-worker rejection, and the proportion of tasks that recover to a verified outcome. A high service uptime percentage can coexist with lost approvals or duplicate transfers.

Not every task needs durable orchestration. A short, read-only request can often fail explicitly and be retried by the user. Durability becomes more valuable as work lasts longer, crosses more dependencies, waits on external events, accumulates verified progress, or can create costly or irreversible effects.

Start with a failure model

Treat failures according to what another attempt can change:

Failure class	Example	Default response
Invalid request or invariant violation	Malformed arguments, illegal transition	Repair the request or fail; do not retry unchanged
Authorization or policy denial	Missing permission, expired approval	Obtain valid authority or escalate; do not reinterpret as transient
Business rejection	Insufficient funds, conflicting booking	Choose another valid path or report the rejection
Transient dependency failure	Rate limit, temporary unavailability	Retry only if the operation is safe and budget remains
Overload	Saturated queue or provider capacity	Apply backpressure, shed load, wait, or degrade before retrying
Partial effect	Two of three requested records changed	Preserve completed effects; continue, reconcile, or compensate deliberately
Unknown outcome	Write timed out after dispatch	Query by operation identity or read authoritative state before another write
Concurrency conflict	State version changed while paused	Reload, revalidate, and replan against the newer version
Cancellation or expiry	User cancels while an operation is in flight	Stop new work, resolve in-flight effects, and enter an explicit terminal or compensating state
Corrupt or incompatible recovery state	Snapshot schema cannot be migrated	Quarantine and escalate rather than guessing or starting over silently

An error code is not enough without semantics. timeout describes the caller’s wait, not whether the server committed. 500 does not guarantee no side effect. rate_limited may be safe to retry later, while permission_denied will not improve through exponential backoff.

Keep infrastructure failures distinct from semantic task failures. Another identical model call may produce a different answer, but randomness is not a recovery strategy. Retry a model decision only when the product can tolerate another sample and the run record makes clear which output was accepted. If new evidence or a different route is required, represent that as a changed strategy, not an invisible transport retry.

Use a durable execution boundary

A reliable runtime separates orchestration state from external work:

durable run record and event history
       │
       ├─ decide the next legal transition
       ├─ record intent, identity, version, limits, and effect key
       ▼
bounded activity: model call, API request, human task, or computation
       │
       ▼
record typed result and authoritative evidence
       │
       └─ update state, wait, retry, compensate, escalate, or finish

The durable record should make accepted transitions reconstructable. External activities may be attempted more than once, so their contracts must carry identity, timeout, retry, and effect semantics. Do not place an unrecorded network call inside logic that will be replayed to rebuild state.

Temporal is one concrete implementation of this separation: workflow state is reconstructed from event history, while Activities perform failure-prone external work and are recommended to be idempotent (Temporal workflow execution; Temporal Activities). OpenAI’s current Agents SDK exposes a serializable RunState as a pause-and-resume boundary and documents integrations with durable orchestrators for runs spanning waits and restarts (OpenAI Agents SDK run state; running agents). These implementations demonstrate the pattern; neither removes the application’s responsibility for external-effect correctness.

A database row and job queue may be sufficient for a small product. A workflow engine becomes attractive when there are many timers, long waits, child tasks, compensation paths, versions, and operational queries. The design requirement is reconstructability and controlled effects, not adoption of a particular platform.

Make retries an explicit policy

A retry is justified only when:

1. the failure class permits another attempt;
2. the operation is read-only, idempotent, or otherwise protected against duplicate effects;
3. the caller still has a valid deadline, authority, and resource budget;
4. another attempt has a credible chance of a different result; and
5. attempts and final disposition remain observable.

For each dependency and operation define:

• retryable and non-retryable outcomes;
• maximum attempts and total elapsed time;
• per-attempt timeout and end-to-end deadline;
• exponential or other backoff and randomized jitter;
• whether Retry-After or provider-specific guidance is honored;
• the single layer that owns retries, or a coordinated budget across layers;
• concurrency and rate limits during recovery;
• idempotency and reconciliation behavior; and
• the state entered when retries are exhausted.

Unbounded retries convert a dependency problem into a queue, cost, and capacity problem. Retries at several layers multiply: four attempts in each of three layers can create 64 attempts at the lowest dependency. Google SRE documents how retries can amplify overload into cascading failure and recommends randomized backoff, bounded per-request attempts, retry budgets, clear error classes, deadline propagation, and avoiding retries at multiple layers (Google SRE, “Addressing Cascading Failures”).

Backoff is not permission to wait forever. Propagate an absolute task deadline so downstream work cannot continue after the user-visible operation has expired. Reserve time for result processing and cleanup. A long-running business task may have a days-long completion deadline while each model or API attempt has a much shorter timeout.

Use backpressure and admission control before retry storms begin. Bound queues, in-flight work, child runs, model calls, and recovery concurrency. During overload, rejecting or delaying lower-priority work can preserve more useful outcomes than attempting everything and completing nothing.

Idempotency represents logical intent

An idempotency key should identify one logical effect, not one network attempt. Reuse it for retries of the same intent; create a new key for a genuinely new intent.

A robust contract includes:

principal and tenant scope
operation name and target
logical intent or effect identifier
normalized request digest or protected version reference
creation and expiry time
status: accepted / running / succeeded / rejected / unknown
canonical response or effect evidence

The receiving service should atomically associate the key with the effect or a durable operation record. A repeated request with the same key and equivalent intent returns the existing result or status. The same key with different material parameters should be rejected, not silently treated as either a duplicate or a new request. Retain deduplication records for at least the credible retry and delayed-delivery horizon.

AWS’s Builders’ Library article recommends caller-provided request identifiers because identical parameters do not reliably mean identical intent; it also emphasizes atomic recording of the token and mutating operation (AWS Builders’ Library, “Making retries safe with idempotent APIs”). The key is only as strong as the service enforcing it. Generating a UUID in the agent harness does not make an unprotected downstream side effect idempotent.

Idempotency does not mean the function executed once. A worker may finish an external write, crash before recording completion, and cause another attempt. The engineering claim is that repeated attempts produce one accepted logical effect or an equivalent observable result. Preserve attempt identifiers separately from the logical effect identifier so operators can see both.

Treat unknown outcome as a first-class state

After a dispatched write times out, three facts may all be true:

• the caller did not receive a response;
• the server may have committed the effect; and
• another attempt may create a duplicate.

The safe next step is usually reconciliation, not guessing:

1. retain the logical effect and attempt identifiers;
2. query an operation-status endpoint or authoritative state;
3. verify whether the observed resource was produced by this intent;
4. wait when the operation is still pending;
5. retry with the same idempotency key only when the contract permits it; and
6. escalate or enter manual review when evidence cannot resolve a consequential effect.

Design interfaces to support this path. A POST that returns no operation identifier and offers no read-back method leaves callers with unsafe choices. Prefer asynchronous operation records, stable resource identifiers, status queries, webhooks, or event correlation when effects may outlive the request.

Do not convert unknown into failed after an arbitrary timeout. Age it, alert on it, and define who owns resolution. Completion remains blocked until required effects are confirmed or the final result explicitly reports the unresolved condition.

Checkpoint truth, not just conversation

A useful checkpoint preserves what another process needs to continue safely:

• run, task, tenant, principal, and ownership identifiers;
• goal and accepted success criteria;
• authoritative state version and completed verified steps;
• pending intents, in-flight attempts, committed effects, and unknown outcomes;
• approvals, their scope and expiry, and pending human decisions;
• model, prompt, policy, context, interface, workflow, and schema versions;
• consumed and remaining budgets, deadlines, timers, retry state, and child-run status;
• artifacts and evidence by durable reference;
• cancellation and compensation state; and
• the exact resume condition and next legal transitions.

A transcript summary or model-generated progress note is a derived view. It can help the next model understand the task, as Anthropic’s long-running coding experiments found with structured progress artifacts and clean incremental handoffs (Anthropic, 2025). It cannot replace authoritative effect, approval, and state records, and Anthropic explicitly presents that work as one coding-oriented harness experiment rather than a general reliability proof.

Write checkpoints at semantic boundaries: after an accepted state transition, before and after a consequential effect, when entering a wait, after receiving an approval, and before handing work to another owner. Checkpointing every token is expensive and usually meaningless; checkpointing only at final output loses too much.

Protect checkpoints like production state. Encrypt sensitive fields, apply tenant isolation and retention rules, validate integrity, and test restore—not merely backup creation.

Resume by revalidating the present

Resumption is not loading bytes and continuing at the next line. Time has passed, so assumptions may no longer hold. Before another material action:

1. validate the checkpoint schema and migrate it through a tested path;
2. reacquire ownership with a lease or equivalent fencing mechanism;
3. reload authoritative state and reject stale writes;
4. reconcile in-flight and unknown external effects;
5. revalidate identity, permissions, policy, approvals, secrets, and data access;
6. check model, interface, prompt, and workflow compatibility;
7. apply user corrections, cancellation, deadlines, and newer events;
8. rebuild bounded model context from current records; and
9. decide whether to continue, replan, compensate, escalate, or terminate.

An approval for one amount, target, document version, or policy state should not silently authorize a changed operation after resume. A model upgrade may improve capability while changing tool selection or output semantics; pin the old configuration where safe, migrate with evaluation evidence, or require a new decision.

Use optimistic concurrency, leases, and fencing tokens so an old worker cannot commit after a newer worker resumes the same run. A lease timeout alone is insufficient: a paused process may wake after losing its lease. The effect boundary must reject its stale fencing value or state version.

Waiting and cancellation are durable states

A run waiting for a person, webhook, schedule, dependency, or rate-limit window should persist a subscription or timer and release compute. It should not repeatedly call a model to ask whether time has passed. Durable timers and events resume the run when a named condition occurs.

For long activities, heartbeats can record liveness and coarse progress, but they are not proof that an external effect is safe to repeat. A heartbeat checkpoint must be durable and semantically sufficient to resume the work it claims to cover.

Cancellation is a request that propagates through owned work; it is not proof that every operation stopped. Define:

• who may cancel and at which scopes;
• whether cancellation is graceful or immediate;
• how child runs, queues, timers, and activities receive it;
• what happens to an operation already committed or in flight;
• which cleanup or compensation must finish despite cancellation;
• how late results are rejected or reconciled; and
• which evidence marks cancellation complete.

Never turn a cancelled run back into running merely because a delayed callback arrived. Record the callback, reconcile its effect, and require a new authorized transition if work should resume.

Compensation is not rollback

Distributed external effects often cannot be rolled back atomically. A compensation is a new domain operation that moves the system toward an acceptable state: cancel a reservation, issue a refund, revoke access, publish a correction, or create a human work item.

For each compensable effect, record:

• the original intent, accepted result, and authoritative evidence;
• the conditions under which compensation is allowed or required;
• the compensating operation, authority, deadline, and idempotency key;
• dependencies and ordering constraints;
• expected residual effects, fees, notifications, or legal records; and
• verification and manual-escalation path.

Compensation can fail and may need its own retries, checkpoints, and alerts. It may not restore the exact original state because concurrent actors have changed the world. Microsoft’s Compensating Transaction pattern likewise warns that compensation is application-specific, eventually consistent, potentially fallible, and not necessarily a reverse-order restoration (Microsoft Azure Architecture Center).

Put irreversible steps after all feasible validation and approval. If no responsible compensation exists, narrow authority, stage the result for review, or make irreversibility explicit to the authorized person before commit.

Recover dependencies without moving the failure

Fallbacks and failover change the system, not just its availability. A new model provider may have different data handling, tool support, latency, or output behavior. A replica may be stale. A restored queue may release a burst of obsolete work.

For every critical dependency, define:

• timeout and end-to-end deadline;
• retry and non-retryable classifications;
• circuit breaking or temporary isolation;
• concurrency, queue, and retry budgets;
• degraded modes and the product promise they preserve;
• failover eligibility, consistency, security, and evaluation requirements;
• recovery ordering, warm-up, and backlog-drain rate; and
• how partial, late, and duplicated results are handled.

Graceful degradation should preserve truth. Returning a narrower answer with an explicit limitation can be reliable; silently replacing verified current data with stale or invented data is not. Fail closed when the missing dependency protects authorization, privacy, or a non-compensable effect.

Recovery traffic needs admission control. After an outage, queued runs, scheduled retries, caches, and clients may all restart together. Add jitter, prioritize time-sensitive work, discard expired work, reconcile before replaying writes, and drain the backlog below the dependency’s safe capacity.

Test recovery as a product behavior

Unit tests for retry helpers are not enough. Exercise the system at every material boundary:

• crash before dispatch, after dispatch, after effect commit, and before result recording;
• drop, duplicate, delay, and reorder messages and callbacks;
• return partial, malformed, stale, and unknown results;
• expire approvals, credentials, policies, deadlines, and leases while waiting;
• resume the same checkpoint concurrently on two workers;
• deploy compatible and incompatible workflow, schema, model, prompt, and interface versions;
• exhaust retries, queues, budgets, storage, and provider capacity;
• cancel during model generation, external write, child work, and compensation;
• fail a compensating action and require manual intervention; and
• restore from backups and drain a realistic backlog without another cascade.

Verify final authoritative state, external effects, evidence, and user-visible outcome—not only whether the worker returned. Feed confirmed failures into the evaluation system and observability alerts. Record recovery time, lost progress, duplicate and unknown effects, retry amplification, and manual effort.

Trade-offs

Choice	Benefit	Cost or risk
Simple job queue and database	Low operational and conceptual overhead	Application owns timers, deduplication, versioning, compensation, and recovery queries
Durable workflow engine	Persisted history, timers, retries, and operational controls	Platform dependency, replay constraints, event-history cost, and migration work
Fine-grained activities	Smaller retry and compensation units	More events, coordination, latency, and schema surface
Coarse activities	Less orchestration overhead	Larger duplicate-effect and restart scope; weaker cancellation and diagnosis
Frequent checkpoints	Less progress loss	More writes, storage, contention, and versioned state
Long deduplication window	Handles delayed retries and callbacks	Storage, privacy, and key-lifecycle cost
Aggressive retries	Hides short transient failures	Cost amplification, overload, duplicate effects, and longer tail latency
Strict fail-fast behavior	Preserves capacity and explicit truth	More visible failures and manual/user retries
Automatic compensation	Faster convergence for clear cases	Wrong compensation can compound harm
Manual recovery	Handles ambiguous consequential cases	Slow, expensive, and dependent on good evidence and staffing

Failure modes

Restart the conversation from the beginning

After a crash, the system asks the model to infer progress from chat history and repeats actions whose effects were already committed.

Retry every exception

Validation, permission, business rejection, overload, and unknown write outcomes all enter the same retry loop. Permanent errors waste resources; ambiguous writes duplicate effects.

Idempotency key per attempt

Every network retry generates a fresh key, so the receiving service correctly treats each attempt as a new effect.

Same key, changed intent

The system reuses a key after changing target, amount, or operation parameters. The service either returns an irrelevant old result or applies a dangerous new request under ambiguous identity.

Timeout means failure

A caller retries a timed-out write without status lookup or read-back. Both attempts commit.

Hidden retries at every layer

SDK, model gateway, service, queue consumer, and workflow each retry independently. A small failure becomes a traffic and cost cascade.

Checkpoint without in-flight effects

The snapshot records the plan and messages but not an already-dispatched operation. Resume repeats it.

Resume stale authority

A run wakes days later with expired approval, changed policy, revoked access, or superseded user correction and continues as if nothing changed.

Lease without fencing

An old worker resumes after its lease expired and overwrites work committed by the new owner.

Compensation as delete-and-restore

Recovery overwrites concurrent legitimate changes or assumes an external effect is perfectly reversible.

Backlog recovery causes another outage

All delayed work and retries resume at once, overwhelming the dependency before normal traffic can recover.

Durable infrastructure, non-durable meaning

The workflow engine replays perfectly, but success criteria, configuration compatibility, approvals, and external effects were never recorded precisely enough to continue safely.

Fieldbook guidance

For each consequential operation, maintain a recovery contract:

Logical intent and effect identifier:
Principal, tenant, target, and preconditions:
Authoritative state version:
Possible success, rejection, partial, and unknown outcomes:
Attempt timeout and end-to-end deadline:
Retryable conditions, owner, attempts, backoff, jitter, and total budget:
Idempotency-key scope, equivalence rule, storage, and retention:
Status query and reconciliation method:
Checkpoint before and after effect:
Cancellation and late-result behavior:
Compensating action, authority, ordering, and verification:
Irreversible point and required approval:
Manual escalation and evidence:

For each durable run type, maintain a resume contract:

Persisted state, event history, artifacts, and schema version:
Checkpoint boundaries and maximum accepted-progress loss:
Timers, subscriptions, child work, and ownership:
Lease, concurrency, and fencing rules:
Configuration pinning and migration policy:
Identity, approval, policy, secret, and state revalidation:
Unknown-effect reconciliation before continuation:
Recovery time and backlog-drain objectives:
Quarantine and manual-recovery path:
Restore and failure-injection test evidence:

If a team cannot state what a retry means, what one logical effect is, and how a new worker distinguishes completed from unknown work, it does not yet have durable execution—it has persistent hope.

Checklist

• Are reliability objectives stated in terms of accepted progress, effects, recovery time, and verified outcomes rather than uptime alone?
• Does every material failure class map to retry, wait, repair, reconcile, compensate, escalate, or terminal behavior?
• Is orchestration state durably separated from replay-unsafe external work?
• Are retryable errors explicit, bounded by attempts and an end-to-end deadline, and owned at one layer or by one shared budget?
• Do retries use backoff, jitter, admission control, and dependency capacity limits?
• Does one idempotency key represent one logical intent across attempts, with mismatched parameters rejected?
• Is the key atomically associated with the effect or durable status and retained for the full retry horizon?
• Can every consequential asynchronous write be queried or reconciled by stable operation identity?
• Are partial and unknown outcomes prevented from becoming silent success or ordinary failure?
• Do checkpoints include authoritative state versions, pending intents, in-flight attempts, effects, approvals, budgets, timers, cancellation, and compatibility metadata?
• Does resume revalidate state, ownership, identity, policy, approval, configuration, deadlines, and newer events?
• Can stale workers be rejected through fencing or version checks after ownership changes?
• Do waits release compute and resume from durable events or timers rather than model polling?
• Does cancellation resolve in-flight and late effects before claiming completion?
• Are compensation steps domain-specific, idempotent, observable, resumable, and able to escalate?
• Are irreversible steps delayed until feasible validation and approval are complete?
• Do failover and degradation preserve security, consistency, and the user promise?
• Is backlog recovery prioritized, rate-limited, jittered, and tested against another cascade?
• Do failure-injection and restore tests verify authoritative state and external effects at every crash boundary?

References and how they are used

• OpenAI Agents SDK, “Run state” and “Running agents” — OpenAI-maintained SDK documentation for the Agents SDK, used for serializable pause/resume state, approval interruptions, compatibility metadata, and durable-orchestrator integrations. These APIs do not guarantee downstream idempotency or application-level recovery correctness.
• Anthropic, “Effective harnesses for long-running agents” — Anthropic engineering experiment used for incremental progress, structured handoff artifacts, and clean cross-session state for long coding tasks. It is coding-specific and explicitly leaves broader long-horizon questions open.
• Temporal, “Workflow Execution”, “Activities”, and “Activity Definition” — Temporal-maintained product documentation used for event-history replay, separation of orchestration from external activities, timeouts, heartbeats, retries, and idempotency. It describes one platform and its exact guarantees should not be generalized to other runtimes.
• AWS Builders’ Library, “Making retries safe with idempotent APIs” — Amazon production engineering article supporting caller-provided intent identifiers, atomic deduplication records, status reconciliation, and semantic equivalence. It reflects Amazon’s service design experience rather than a formal interoperability standard.
• Google SRE, “Addressing Cascading Failures” — Google SRE chapter supporting deadlines, bounded retries, randomized backoff, retry budgets, load shedding, and avoiding retry multiplication. Its scale examples require adaptation, but the feedback-loop risks apply broadly.
• Microsoft Azure Architecture Center, “Compensating Transaction pattern” — Microsoft Azure Architecture Center pattern article used to distinguish compensation from rollback and to cover concurrency, failure, idempotency, irreversible points, and human intervention. The Azure example is illustrative rather than required.