Cost, Latency, and Capacity

An inexpensive model call can belong to an expensive system. A fast first token can precede a slow or failed task. A service with low average utilization can still collapse under long-tail work, bursts, retries, or a constrained dependency. Optimizing one provider metric rarely optimizes the outcome a user needed.

Agentic systems make this problem more visible because one accepted task can create an unpredictable number of model calls, retrievals, tool operations, approvals, retries, artifacts, and follow-up checks. The work graph can change while the task runs. Costs and service times are distributions, not constants.

Thesis

Manage cost, latency, and capacity as one constrained operating problem. Optimize the cost and time of accepted outcomes—not requests or tokens—under explicit quality, reliability, safety, privacy, fairness, and service requirements. Expose the full work graph, control demand before saturation, reserve capacity for important work, and verify every optimization on representative outcomes and tails.

This does not mean every team needs a financial model or queueing simulator before a prototype. It means the unit, boundary, and constraints must become explicit before scale or consequence makes a misleading metric expensive.

Define the unit before optimizing it

Start with the smallest unit that represents delivered value. Examples include a support case resolved without an incorrect external action, a code change accepted after tests and review, or a research brief with required citations verified. Call this an accepted outcome: a result that meets the task’s named completion, quality, policy, and effect criteria.

Then measure:

cost per accepted outcome =
  total attributable cost for the workload
  / number of accepted outcomes

accepted throughput =
  accepted outcomes / unit time

acceptance yield =
  accepted outcomes / admitted tasks

A request, call, or token remains useful for diagnosis and allocation, but it is a resource unit rather than proof of value. The FinOps Foundation makes the same distinction between resource-efficiency units such as cost per token and business units such as cost per case resolved (FinOps Foundation, Unit Economics).

Define “accepted” before using the denominator. If the system quietly counts every generated answer as a success, a cheaper model can appear efficient by producing more unusable results. If humans correct outputs, either count the correction cost and final acceptance or classify the original attempt as not accepted. Do not hide abandoned, duplicate, blocked, or policy-violating work.

For an early experiment where outcome attribution is not yet possible, use a ladder rather than pretending precision:

1. provider cost per call or token;
2. total technical cost per admitted task;
3. total cost per completed task;
4. total cost per accepted outcome; and
5. cost relative to outcome value, risk reduction, or an alternative process.

Record which rung supports the current decision and what it omits.

Map the full cost of work

Build a cost tree for one logical task. Depending on the system, it can include:

Cost group	Examples
Model work	uncached and cached input, cache writes, generated output, reasoning, embeddings, reranking, moderation, evaluator calls
External operations	search, retrieval, browsers, code execution, paid APIs, messages, transactions, data transfer
Application infrastructure	compute, accelerators, databases, queues, object and vector storage, networking, secrets, orchestration
Operating evidence	logs, traces, metrics, retention, redaction, sampling, evaluation suites, load tests
Human work	approval, review, correction, escalation, support, incident response, compliance and domain assurance
Waste and recovery	failed attempts, retries, speculative branches, duplicate effects, abandoned work, cache misses, replay and repair
Shared and fixed costs	engineering, licenses, commitments, reserved capacity, security, governance, unused minimum capacity

Do not allocate every shared cost perfectly before acting. First expose the large variable drivers and the costs that could reverse a decision. Add fully loaded costs when comparing products, providers, self-hosting, staffing, or long-term viability.

Provider usage fields are inputs to this tree, not the tree itself. Tokenization and accounting can change by model. Anthropic, for example, explicitly advises recounting prompts against the target model rather than reusing counts from an earlier tokenizer (Anthropic token counting). Keep raw provider usage, price version, currency, discounts, and allocation rules so historical costs can be recomputed.

Attribute cost through the work graph

Give every task and child operation stable correlation identifiers. Attribute each cost event to at least:

• logical task and attempt;
• user, tenant, product, and workload class where permitted;
• model, provider, region, service tier, prompt and policy version;
• operation type, cache status, tool or dependency;
• accepted, rejected, cancelled, expired, duplicate, or unknown outcome; and
• whether the work was on the final critical path, useful off-path evidence, or discarded.

Shared infrastructure can be allocated with a documented driver such as occupied time, calls, bytes, reserved capacity, or accepted outcomes. There is no universally correct driver. Preserve direct costs separately so a changing allocation formula does not rewrite operational facts.

Measure end-to-end latency, not one API span

Define start and end events for each workload class. Interactive work might begin when the service admits a user request and end when a usable response is displayed. A background task might end only when its artifact is committed, checks pass, and the requester is notified.

Decompose the elapsed time:

end-to-end latency =
  admission and queue wait
  + context and input preparation
  + model processing and generation
  + sequential control decisions
  + retrieval and tool execution
  + human or external waits
  + retries, recovery, and persistence
  + final validation and delivery

Report at least three different views when relevant:

• Time to first useful response — when the consumer first receives task-relevant, usable information.
• Time to completion — when the defined terminal outcome is available.
• Active processing versus waiting — where elapsed time was actually spent.

Time to first token (TTFT) is a useful provider and streaming measurement, as Anthropic’s latency documentation describes, but a token is not necessarily useful (Anthropic, Reducing latency). A quickly streamed preamble can improve TTFT while delaying the answer. Streaming changes when information is visible; it does not by itself reduce total work.

Measure tails and timeout treatment

Means hide the cases that dominate user frustration, resource occupation, and retry pressure. For each workload class, record a distribution such as p50, p90, p95, and p99, plus:

• the measurement window and population;
• whether queue time is included;
• the share timed out, cancelled, rejected, or still running;
• cold starts, cache hits and misses, and dependency state;
• input size, output size, tool count, and model configuration; and
• accepted-outcome rate for the same cohort.

Do not remove timed-out work from the latency distribution and report only successful requests. Report the conditional success latency and the timeout or non-completion rate together. A release that makes successful calls faster by timing out difficult tasks earlier is not automatically an improvement.

Optimize the critical path

Draw the work as dependencies rather than as a flat list of calls. The critical path is the longest dependent sequence that determines completion time. Reduce work there first.

OpenAI’s current latency documentation groups practical techniques into processing faster, generating fewer tokens, using fewer input tokens, making fewer requests, parallelizing, reducing user waiting, and avoiding an unnecessary model call (OpenAI, Latency optimization). The durable lesson is to inspect the work graph, not to apply every technique mechanically.

Remove unnecessary sequential decisions

Each serial model or dependency call adds round-trip and queueing delay. Combine steps when one decision can safely produce a structured result. Use deterministic code for exact lookup, validation, formatting, arithmetic, or policy checks that do not benefit from model interpretation. Do not split work merely to route every small step to a cheaper model; added calls can erase the saving.

Parallelize only independent work

Parallel execution can shorten a critical path when branches do not depend on one another. It also increases instantaneous concurrency, quota use, fan-out, cancellation complexity, and the probability that at least one dependency is slow. Bound the number of branches, cancel obsolete work, and measure the total work consumed per accepted outcome.

Speculative execution is a deliberate exchange: spend extra capacity now in the hope of shortening a likely path. Use it only when the probability, saved latency, cancellation behavior, and waste budget justify the extra work. Never let speculative operations commit external effects.

Reduce input and output with evidence

Context selection belongs to context engineering; this chapter measures its consequences. Remove duplicate or irrelevant material, cap outputs to what the consumer needs, and avoid verbose intermediate model text that no component uses. Validate quality after every reduction. A short context that omits decisive evidence is more expensive when it causes correction or harm.

Use the cheapest eligible configuration, not the cheapest call

Model Strategy and Routing defines which configurations are eligible for a task. Runtime routing may choose among them using current quality, latency, cost, availability, and capacity signals. It must not route outside the evaluated envelope simply to meet a spend target.

OpenAI’s cost-optimization documentation explicitly connects smaller models and fewer tokens or requests with latency and cost, while preserving accuracy as a constraint (OpenAI, Cost optimization). Anthropic similarly advises establishing a quality baseline before applying latency reductions (Anthropic, Reducing latency). The comparison that matters locally is cost and latency per accepted result, including retries and review.

Establish workload classes and budgets

One global timeout, queue, priority, or model budget is rarely coherent. Define workload classes from consumer need and consequence, for example:

Workload class	Typical objective	Likely treatment under pressure
Interactive	useful first response quickly; bounded completion	reserve low-latency capacity, cap fan-out, stream useful progress, reject or defer before deadlines become impossible
Transactional	predictable completion and effect correctness	preserve validation and effect controls; queue briefly or fail clearly rather than silently degrade correctness
Long-running	durable progress within a broad deadline	checkpoint, schedule fairly, pause or resume, enforce per-task and cumulative budgets
Batch or evaluation	complete a volume by a deadline	use asynchronous capacity, fill valleys, preempt or delay for higher-priority work
Emergency or control-plane	preserve containment and recovery	isolate and reserve capacity; do not let ordinary tenant load starve it

For each class define:

• admission rule and maximum queue or deferral time;
• time-to-first-useful-response and completion objectives where applicable;
• quality, reliability, safety, and privacy invariants;
• per-task model, token, tool, wall-time, and spend budgets;
• concurrency, rate, and cumulative tenant or user limits;
• retry, branch, and escalation budgets;
• priority and fairness rules;
• degradation, cancellation, expiry, and notification behavior; and
• capacity owner, forecast horizon, and review triggers.

A service level objective (SLO) is a measured target, not an aspiration. Start from behavior the consumer cares about and state the valid population and measurement method, consistent with Google SRE’s SLO chapter (Google SRE, Service Level Objectives). Quality and policy constraints belong beside latency and availability objectives, even if they use different evidence.

Relate demand, time, and capacity

Track four different rates:

• offered demand — work clients attempted to submit;
• admitted demand — work the system accepted;
• completed throughput — work that reached a terminal state; and
• accepted throughput — completed outcomes meeting acceptance criteria.

The gaps reveal shedding, expiry, failure, or quality loss. A system can maintain raw completion throughput while accepted throughput falls.

For a stable population, Little’s Law gives a useful first-order relation:

average work in system ≈ arrival rate × average time in system
N ≈ λW

John D. C. Little proved the general queueing relation in 1961 (Little, “A Proof for the Queuing Formula: L = λW”). If 2 tasks per second spend an average of 20 seconds admitted but unfinished, expect about 40 tasks in flight. If service time doubles while arrivals remain constant, concurrency and the resources held by those tasks tend to double.

This is not a capacity guarantee. It uses averages and does not predict bursts, high percentiles, priorities, finite queues, correlations, or an unstable system whose backlog grows without bound. Use distributions and load tests for those.

Find the real bottleneck and the saturation knee

Capacity is limited by the first constrained resource across the whole work graph, which may be:

• provider requests, input tokens, output tokens, concurrent jobs, or spend;
• application workers, CPU, accelerators, memory, connections, file descriptors, or storage I/O;
• queue partitions, retrieval services, browsers, sandboxes, third-party APIs, or identity services;
• human review, approval, support, or domain expertise; or
• tenant-specific, regional, contractual, security, or data-residency limits.

Increase offered load in a controlled environment and plot accepted throughput, latency percentiles, queue age, errors, retries, and resource use. The saturation knee is where added load begins to create disproportionate delay, failure, or reduced useful throughput. Set normal operating targets below it with headroom for bursts, failover, maintenance, model variance, and forecast error.

Provider limits are not promised capacity. Anthropic documents request, input-token, and output-token limits, short-window burst enforcement, and acceleration limits for sharp increases (Anthropic rate limits). OpenAI likewise warns that failed retries still consume per-minute capacity (OpenAI rate limits). Measure the actual dependency and negotiate or diversify capacity before a launch, not during saturation.

Control demand before overload controls you

Autoscaling is useful but not instantaneous and cannot create a constrained external dependency. Layer the controls.

Bound work at admission

At the earliest informed boundary, decide whether to accept, start, queue, defer, downgrade, or reject. Consider workload class, deadline, estimated work, current saturation, tenant allocation, dependency health, and whether the system can still finish before the result loses value.

Admission is a promise to spend capacity, not merely a queue insertion. Rejecting early with a clear retry or deferral contract is often better than accepting work that will expire after consuming resources.

Propagate backpressure

When downstream capacity is constrained, reduce upstream creation of new work. Stop polling faster, pause batch producers, reduce branch creation, return retry guidance, and cap client concurrency. If backpressure ends at an internal queue while clients continue submitting, the queue only moves the overload.

Google’s client-side throttling experience shows why rejecting closer to the producer can avoid consuming backend resources merely to reject the same request later (Google SRE, Handling Overload). Implementations differ, but the direction of the signal should be explicit.

Keep queues bounded and deadline-aware

Queues smooth short bursts and decouple asynchronous work, but they consume memory, hide demand, and add latency. Record queue depth, oldest-item age, deadline slack, class, tenant, and cancellation state. Bound both items and estimated work, because ten large agent tasks can exceed a thousand small requests.

Google SRE notes that queued work increases latency and can conceal saturation; queue size should match measured burst and service behavior rather than a large comforting number (Addressing Cascading Failures). Expire work that can no longer meet its objective, and make expiry visible to the caller.

Enforce priority and fairness

Priority answers which work matters first; fairness prevents one source from consuming every shared resource. Use per-tenant quotas, weighted scheduling, concurrency partitions, aging, or reserved pools as the workload requires. Protect control-plane and recovery work from ordinary traffic.

Test starvation and priority inversion: low-priority work may hold a scarce worker, lock, connection, or human reviewer required by high-priority work. Admission priority without resource isolation may provide no real guarantee.

Shed excess load deliberately

Load shedding means intentionally rejecting, deferring, or cancelling some excess work so the system can still complete the most useful admitted work. Shed before the service consumes the resources needed to reject, times out every client, or enters a retry feedback loop.

Possible order, subject to product policy, is:

1. reject invalid, duplicate, expired, or unauthorized work;
2. stop speculative and optional branches;
3. pause deferrable batch work;
4. restrict expensive optional features;
5. enforce tenant allocations and shed lower-priority excess;
6. reject new work whose deadline cannot be met; and
7. preserve recovery, notification, and control paths.

Amazon and Google production accounts both emphasize that overload can reduce useful throughput and amplify itself through queues and retries, and that shedding must be observable and tested (Amazon Builders’ Library; Google SRE). Do not invent a complex degraded path and wait for an incident to exercise it.

Degrade only within an approved envelope

Graceful degradation may reduce search breadth, freshness, optional enrichments, output detail, or model cost. It must preserve the class’s safety, authorization, correctness, and disclosure requirements and tell consumers when the result is materially reduced.

Some workloads must defer or fail closed. A cheaper unevaluated model, stale cache, omitted approval, or incomplete compliance check is not graceful merely because the endpoint returned 200.

Treat caching as a correctness and economic decision

Caching can avoid repeated work, reduce latency, and expand effective throughput. It can also serve stale or cross-scope data, obscure provenance, create write and storage cost, and fail when a provider changes matching rules.

Distinguish at least:

Cache	Reused object	Main correctness question
Provider prompt or prefix cache	computation for an exact input prefix	Are matching, lifetime, isolation, price, and rate-limit semantics understood for this provider and model?
Exact application cache	a previously computed result for an identical key	Does the key include every input, policy, model, tenant, authorization, and version that can change validity?
Semantic cache	a result judged reusable for a similar request	Is similarity sufficient for this task, and are scope, freshness, provenance, and uncertainty enforced?
Retrieval or tool cache	external data or operation result	What is the authoritative freshness rule, and can the operation have user-specific or changing effects?

For each cache define:

• key and matching semantics;
• tenant, principal, purpose, region, and data-class isolation;
• source and transformation provenance;
• freshness, invalidation, and deletion rules;
• negative-result and error caching behavior;
• write, read, storage, lookup, miss, and stale-result costs;
• bypass, rollout, rollback, and observability; and
• evaluation of hit correctness, not only hit rate.

Provider prompt caches illustrate why implementation details must be dated. OpenAI currently caches eligible exact prefixes, exposes cache-write and read usage, and documents organization isolation and model-family-specific retention and pricing (OpenAI prompt caching). Anthropic exposes cache creation and read tokens, configurable lifetimes, invalidation rules, isolation, and cache-aware rate limits (Anthropic prompt caching). These are important current product references, but neither defines application-cache correctness and both can change.

A cache is economically useful only when:

expected avoided work from valid hits
  > write + storage + lookup + invalidation + miss + stale-result cost

Segment hit rate by reusable prefix or key, model, tenant, workload, and time gap. A high aggregate hit rate can hide a critical workload with no benefit or incorrect reuse.

Separate interactive, asynchronous, and batch capacity

Not every task needs immediate execution. Asynchronous processing can trade completion time for lower price, smoother utilization, and separate capacity. Both OpenAI and Anthropic currently offer discounted batch processing for work that accepts a broad completion window (OpenAI Batch API; Anthropic batch processing). Treat the discount and window as dated product behavior; preserve the general design question.

Batch or deferred work still needs:

• a deadline and maximum queue age;
• input, policy, model, and price version capture;
• per-item identity, idempotency, cancellation, and result status;
• partial-failure and replay policy;
• reserved capacity or preemption rules;
• notification and result retention; and
• evaluation of whether delayed data remains valid.

Batching can improve throughput by amortizing overhead or using cheaper capacity, but it adds waiting, larger failure domains, and delayed feedback. Do not batch interactive work merely because one provider call is cheaper.

Forecast and test capacity with realistic work

A capacity plan should state:

forecast horizon and confidence range
workload classes and arrival distributions
task-size and service-time distributions
accepted-outcome and abandonment assumptions
peak, burst, event, growth, and regional patterns
cache hit/miss and warm/cold assumptions
model, provider, tool, human, and infrastructure limits
failure, maintenance, failover, and recovery reserve
planned headroom and trigger for adding or shifting capacity

Do not forecast from monthly token totals alone. Two workloads with the same tokens can have different request rates, outputs, tool waits, parallelism, and concurrency. Forecast by workload shape and reconcile the resulting resource demand with provider limits, infrastructure, people, and spend.

Load-test the work graph

Use production-shaped, privacy-safe tasks and controlled synthetic cases. Include:

• ordinary mix plus large inputs, long outputs, and many tool steps;
• cold starts, cache misses, cache churn, and hot-key concentration;
• bursts, gradual ramps, tenant concentration, and priority mixes;
• slow, failing, rate-limited, and unavailable dependencies;
• retry, cancellation, timeout, and unknown-outcome paths;
• long-running tasks spanning deploys or worker loss;
• load above the saturation knee to verify shedding and recovery; and
• a return to normal load to detect stuck queues or leaked concurrency.

Measure offered, admitted, completed, and accepted throughput; latency tails; queue age; saturation; rejection reason; cost; waste; quality; and external-effect correctness. A test that stops when the first latency objective fails does not show whether overload behavior is safe.

Provider-hosted models add shared-service variance. Record observation time, tier, region, model snapshot, limits, and response headers where available. Repeat tests and avoid publishing one transient measurement as a permanent model ranking.

Optimize through controlled changes

Treat an optimization as a change to the evaluated system. Before rollout, write:

Workload and accepted-outcome definition:
Baseline configuration and observation window:
Hypothesis and changed mechanism:
Quality, reliability, safety, privacy, and fairness invariants:
Cost boundary and allocation method:
Latency start/end events and required percentiles:
Capacity and overload assumptions:
Expected effect and uncertainty:
Evaluation, load test, and rollout plan:
Rollback and invalidation triggers:
Owner and review date:

Compare the same workload distribution or explicitly reweight it. Report confidence intervals or repeated-run variation where model behavior is material. Use staged rollout and watch accepted outcomes, not just provider spend.

An optimization is suspect when it:

• lowers cost per call but raises attempts per accepted outcome;
• improves p50 while worsening p99, timeouts, or abandonment;
• raises raw throughput while reducing acceptance yield;
• increases cache hits while serving stale or cross-scope results;
• moves work off the measured critical path into hidden human correction;
• satisfies an average budget by starving a tenant or important class;
• converts clear rejection into a long queue; or
• uses an unevaluated model or drops a required control.

Common failure modes

Token tunnel vision

The team optimizes token price while tool calls, retries, review, or failures dominate total cost. Response: publish the cost tree and cost per accepted outcome beside token metrics.

Fast first token, slow useful result

Streaming starts immediately but the answer, artifact, or committed effect arrives late. Response: measure first useful response and completion from the consumer boundary.

Parallelism explosion

Every task fans out, shortening some runs while saturating provider quotas and multiplying discarded work. Response: cap branches and global concurrency, cancel obsolete work, and measure useful work per outcome.

Retry amplification

Timeouts trigger retries while original attempts still consume capacity. Response: use the reliability chapter’s bounded retry and idempotency rules, apply backpressure, and include attempts in demand accounting.

Queue as a hiding place

The endpoint accepts everything and reports availability while queue age exceeds task value. Response: bound queues, expose age and deadlines, and reject or defer honestly.

Average-capacity planning

Monthly averages appear safe, but bursts, long tasks, or output-token limits saturate the service. Response: model distributions and each constrained dimension; test peaks and tails.

Cache without a validity model

The team maximizes hit rate without encoding tenant, authorization, policy, freshness, or version. Response: make validity part of the key and test incorrect-hit rate.

Autoscaling as the only defense

Workers scale after a queue forms, while provider, database, or human-review capacity remains fixed. Response: combine headroom, admission, backpressure, quotas, and shedding with scaling.

Cheap degradation that violates the contract

Under pressure the system switches to a model or data path that has not met the workload’s acceptance criteria. Response: predefine and evaluate degradation envelopes; otherwise defer or fail closed.

Unpriced observability cuts

Telemetry is removed to save storage, making waste, tail behavior, and incidents invisible. Response: optimize collection by decision value, sampling, redaction, and retention rather than eliminating necessary evidence.

Practical checklist

Unit and boundary

• Is an accepted outcome defined with completion, quality, policy, and effect criteria?
• Are request, token, completed-task, and accepted-outcome metrics clearly distinguished?
• Does the cost boundary include material tools, infrastructure, retries, waste, evaluation, and human work?
• Can usage and price versions be recomputed historically?

Latency and work graph

• Are start and end events defined for each workload class?
• Are first useful response, completion, queue time, and active processing measured separately?
• Are tail percentiles reported with timeouts, cancellation, and acceptance rate?
• Is the critical path known, and are serial calls justified?
• Is parallel and speculative work bounded and cancellable?

Capacity and overload

• Are offered, admitted, completed, and accepted throughput visible?
• Are service-time and task-size distributions measured rather than assumed constant?
• Is each provider, infrastructure, tool, human, tenant, and regional bottleneck recorded?
• Has the saturation knee been measured with realistic load?
• Is headroom tied to bursts, failover, maintenance, growth, and forecast error?
• Are admission, backpressure, queue, fairness, and load-shedding rules explicit and tested?
• Is control-plane and recovery capacity protected?

Caching and deferred work

• Does every cache have matching, isolation, freshness, invalidation, deletion, and provenance rules?
• Are cache writes, reads, misses, storage, stale results, and incorrect hits measured?
• Are provider cache assumptions tied to a dated model and documentation version?
• Do asynchronous and batch jobs have deadlines, item identity, cancellation, partial-failure handling, and notification?

Change decision

• Is every cheaper or faster configuration still inside the evaluated eligibility envelope?
• Does the comparison use representative workload mix and repeated trials?
• Are quality, reliability, safety, privacy, and fairness invariant or explicitly reapproved?
• Are rollout, rollback, owner, budget, and review triggers recorded?

References

• OpenAI, “Latency optimization” — OpenAI-maintained API documentation for OpenAI workloads, used to examine tokens, requests, parallelism, user waiting, and unnecessary model calls. It does not define end-to-end application SLOs.
• OpenAI, “Cost optimization” — OpenAI-maintained API documentation connecting OpenAI requests, tokens, model choice, batch, and flexible processing to cost and latency. It does not account for full task or human cost.
• OpenAI, “Prompt caching” — OpenAI-maintained API documentation for exact-prefix caching, cache accounting, isolation, retention, and model-family-specific economics. All numeric behavior should be treated as dated.
• OpenAI, “Batch API” — OpenAI-maintained API documentation for exchanging interactivity for discounted asynchronous processing and separate capacity in OpenAI’s platform. Price and limits are not generalized.
• OpenAI, “Rate limits” — OpenAI-maintained API documentation for multi-dimensional quota management and bounded backoff on OpenAI’s platform; provider tiers and enforcement can change.
• Anthropic, “Reducing latency” — Anthropic-maintained product documentation for Claude calls, used for TTFT, model choice, token length, streaming, and quality-first optimization. It does not cover complete agent runs.
• Anthropic, “Prompt caching” — Anthropic-maintained product documentation for Claude prompt caching, used for cache accounting, lifetime, invalidation, isolation, and effective throughput. Details vary by model and platform.
• Anthropic, “Batch processing” — Anthropic-maintained product documentation for discounted asynchronous Claude work; current pricing and supported models are volatile.
• Anthropic, “Rate limits” — Anthropic-maintained product documentation for request and token dimensions, short-window bursts, acceleration, spend limits, and cache-aware capacity on Anthropic’s platform.
• Anthropic, “Token counting” — Anthropic-maintained product documentation showing that token counts depend on the target Claude model and tokenizer.
• FinOps Foundation, “Unit Economics” — Linux Foundation project material for connecting technical resource units to outcome and business units, documenting definitions, and maintaining allocation evidence.
• John D. C. Little, “A Proof for the Queuing Formula: L = λW” — peer-reviewed primary proof of the average queueing relation used for first-order concurrency reasoning; it does not predict tails or unstable overload.
• Google SRE, “Service Level Objectives” — Google SRE chapter for consumer-relevant, explicitly measured service objectives.
• Google SRE, “Handling Overload” — Google SRE chapter for client-side throttling, quotas, and reducing rejected work at the backend.
• Google SRE, “Addressing Cascading Failures” — Google SRE chapter for queue management, overload feedback, load shedding, graceful degradation, and exercising rare paths.
• Amazon Builders’ Library, “Using load shedding to avoid overload” — Amazon engineering article covering saturation, retry amplification, bounded work, priority, visibility, and overload testing; its implementation choices are not universal defaults.