Uncertainty and Variable Model Behavior

Traditional software is not free of uncertainty. Requirements can be ambiguous, networks fail, clocks drift, concurrent writes race, and external services change. Even so, engineers generally expect a deterministic component under controlled conditions to produce the same result from the same state and input.

When a generative model interprets a request, produces an artifact, or helps choose an action, that expectation no longer describes the whole system. The request may support several interpretations, the available evidence may be insufficient, several outputs may be equally useful, repeated runs may take different paths, and tools may observe a changing world. A fluent answer can conceal any of these conditions.

This does not make reliable software impossible. It changes what reliability must mean and where the system must enforce it.

Thesis

Software can remain reliable when model behavior varies, provided the team defines what outputs, processes, and outcomes are acceptable; enforces hard constraints outside the model; and represents unresolved uncertainty explicitly instead of forcing a specific answer.

The goal is not to make every output identical. It is to determine which differences are useful, which are harmless, and which violate the product requirements. That requires engineers to locate uncertainty before selecting a response.

Separate uncertainty, variability, non-determinism, and error

In the strict computational sense, a deterministic computation produces the same result from the same fully specified state and input. Non-determinism means that the same fully specified conditions do not guarantee one result or execution path. Random sampling is one cause; concurrent scheduling can be another.

Product teams rarely observe or control every relevant condition. Model versions, prompts, hidden context, retrieval results, tool responses, time, and concurrent activity may differ between runs that look identical at the product level. That makes exact reproduction difficult, but it does not make all of these differences examples of non-determinism in the strict sense.

Four related terms should remain separate:

Uncertainty is incomplete knowledge about which interpretation, claim, behavior, or environmental state is correct or will occur.
Variability is an observed difference among outputs, trajectories, or outcomes.
Non-determinism is one possible cause of variability, not a general name for ambiguity, missing evidence, or changing external state.
Error means that behavior fails the stated acceptance criteria.

A system can vary without being wrong. It can also be perfectly repeatable and consistently wrong. Exact reproducibility and reliability are therefore different properties. Because this chapter covers all four, it does not use “non-determinism” as an umbrella term.

Four practical questions

The following questions are not a statistical taxonomy. They are a review aid for locating a problem before deciding what the system should do next.

Area to inspect	Diagnostic question	What can make it visible	Typical response
Request	Does it permit materially different interpretations?	Parsed assumptions, missing-field checks, clarification records	Clarify, expose assumptions, or constrain the task
Evidence	Is required evidence missing, conflicting, stale, or unavailable?	Citations, retrieval coverage, source dates, conflict flags	Retrieve, verify, state that the answer cannot be determined, or escalate
Repeated behavior	Can comparable runs produce materially different outputs or trajectories?	Repeated trials, outcome grading, trajectory comparison	Define acceptance criteria, sample deliberately, and validate
Execution environment	Can tools, external state, concurrency, or infrastructure change observations or effects?	Versioned configuration, state snapshots, tool results, timestamps	Isolate runs, enforce idempotency, re-read state, and recover

The areas interact. An ambiguous request can lead to different plans; different plans can retrieve different evidence; changing evidence can lead to different actions. The purpose is not to assign every failure to exactly one box. It is to stop the word “uncertainty” from concealing several different engineering problems.

Input ambiguity

Input ambiguity exists when more than one materially different interpretation remains plausible. “Summarize this for leadership” leaves audience, decision, length, and risk tolerance unstated. “Book a reasonable flight” leaves price, schedule, baggage, and changeability unresolved.

The system should not silently collapse consequential ambiguity into one answer. It can ask a question, state an assumption, present bounded options, or apply an explicit product default. The right choice depends on the cost of interruption versus the cost of acting on the wrong interpretation.

Missing or conflicting evidence

This problem exists when the system lacks sufficient grounds for a claim or decision. Relevant information may be absent from context, unavailable to the model, contradictory across sources, stale, or inaccessible.

Retrieval can reduce this uncertainty, but it does not eliminate it. Retrieved material can itself be incomplete, outdated, irrelevant, or adversarial. Evidence therefore needs provenance, dates, and a defined relationship to the claim it supports. NIST describes confident false or unsupported content as a central generative-AI risk and places validation within a wider lifecycle risk-management process (NIST, 2024).

Run-to-run variability

Run-to-run variability appears when repeated runs produce different wording, answers, plans, tool selections, or outcomes. Some of it is intentional: multiple useful summaries or designs may exist. Other variation reveals fragile requirements: a router changes destinations, an extraction changes totals, or an agent alternates between success and a destructive action.

One successful run says little about the distribution. Anthropic’s agent-evaluation article recommends multiple trials, outcome and trajectory inspection, and environmental isolation because both model behavior and the surrounding harness can introduce variation (Anthropic, 2026).

A changing execution environment

Environmental uncertainty comes from everything a model-assisted run observes or changes outside the model: web pages, inventory, permissions, repositories, queues, other users, clocks, and services. Two apparently identical requests may not be equivalent because the world changed between them.

The response is conventional distributed-systems engineering applied with extra care: snapshot inputs where possible, record versions, use idempotency keys for side effects, validate preconditions immediately before commit, and make partial execution recoverable. A model retry must not silently become a duplicated payment, message, or deployment.

Testing the model against four kinds of work

The four sources remain useful across tasks with very different notions of correctness.

Work	Where uncertainty appears	Acceptance criteria and allowed variation	Hard constraints and response
Invoice extraction	A blurred digit is ambiguous; tax rules or vendor identity may be missing; repeated parses may disagree; OCR or exchange-rate services may change.	Required fields satisfy a schema and confidence rules; values trace to document regions.	Line items must reconcile with totals; currency must be explicit; unresolved high-impact fields go to review.
Research answer	Scope is underspecified; evidence is incomplete or conflicting; runs choose different sources; the web changes.	Claims answer the stated question, citations support them, and material disagreement is disclosed.	No unsupported high-impact claim; access dates and source provenance are recorded; lack of evidence permits abstention.
Creative generation	Audience and tone may be unclear; brand facts may be missing; output diversity is intentional; asset services may vary.	Several outputs can be valid if they meet the brief, policy, factual, and format constraints.	Required facts and prohibited content are checked outside stylistic judgment; exact string equality is not a goal.
Purchasing agent	“Best” depends on unstated preferences; stock and policy change; plans and sellers vary; price can move before purchase.	The selected item meets explicit preferences and organization policy.	Authorization, budget, supplier allowlist, transaction idempotency, and final price are enforced before commit; otherwise pause or escalate.

This comparison exposes an important asymmetry: creative variability may be product value, while financial variability may be product risk. “Turn down temperature” is not a general solution. The requirements must state what may vary in each use case.

Define what counts as acceptable

A deterministic function is often tested against one expected result. A capability that uses a generative model often needs a set of acceptable results. That set should still be explicit and testable.

Define acceptance criteria at three levels:

Output: Is the artifact valid, grounded, complete enough, and within policy? A schema is useful but rarely sufficient.
Process: Did the system use permitted data and tools, respect budgets, preserve provenance, and obtain required approval?
Outcome: Did the task achieve the intended effect without an unacceptable side effect?

An output can look correct while the process violates privacy. A trajectory can follow every permitted step while the final answer is wrong. An outcome can be successful once because the environment happened to compensate for a defect. Reliability requires all material levels, not whichever one is easiest to grade.

Acceptance criteria may include hard rules, graded quality thresholds, and human judgment. Not every criterion needs to be executable in code. The team does need to explain what counts as success, what evidence supports that judgment, and what happens when the evidence is inconclusive.

Enforce hard constraints outside model judgment

In software engineering, an invariant is a condition that must remain true. For a model-assisted capability, requirements involving authorization, money, privacy, or irreversible actions should not depend on the model remembering or interpreting a prompt correctly. Examples include:

the caller is authorized for the requested action;
a transaction never exceeds an approved amount;
structured data satisfies its schema and reconciliation rules;
destructive operations require a valid approval token;
a side effect is applied at most once; and
every externally stated factual claim has traceable support when the use case requires it.

Prompts can tell a model about these rules, but prose is not enforcement. Authorization checks, policy engines, schemas, transaction constraints, allowlists, and idempotency mechanisms should reject invalid actions independently. The model can propose; the application decides whether the proposal is admissible.

Not every requirement can be checked mechanically. “A medical answer must not omit a material contraindication” may need a specialist grader or human review. It still belongs in the acceptance criteria, with an enforcement method proportional to consequence.

Represent unresolved uncertainty explicitly

Many systems offer only success or error, so uncertainty is forced into one of them. If a specific answer is required for success, the system is rewarded for returning one even when the evidence is insufficient; alternatively, a useful partial result may be discarded as failure.

Use explicit states such as:

needs_clarification — the request has materially different interpretations;
insufficient_evidence — the system cannot support a required claim;
conflicting_evidence — credible sources disagree;
needs_approval — a valid proposal crosses a consequence boundary;
retryable_environment_error — the model’s proposal may be sound, but execution conditions failed;
needs_human_review — available automated checks cannot resolve the risk; and
completed_with_assumptions — the result is usable only under disclosed assumptions.

The names will differ by product. The principle is stable: unresolved uncertainty should be recorded explicitly, visible in telemetry, testable in evaluation, and actionable in the interface. This does not require every product to expose these internal names to users.

This changes incentives. Research on hallucination argues that accuracy-only scoring can reward guessing and that evaluations should give models credit for appropriate uncertainty or abstention (Kalai et al., 2025). A product that punishes every unanswered case will train both people and models to conceal uncertainty.

Match the response to the problem

Uncertainty is not one condition, so there is no single “confidence threshold” that resolves it.

Response	Use when	Do not confuse it with
Clarify	A missing user choice would materially change the result.	Asking the user to solve facts the system could retrieve or validate.
Retrieve	Better external evidence is likely available.	Proof that the retrieved evidence is complete or trustworthy.
Validate	A candidate can be checked against a schema, rule, simulator, tool, or independent evidence.	Asking the same model to approve its own answer without calibration.
Abstain	Required support is unavailable or expected harm exceeds the value of a guess.	A generic failure message with no recovery path.
Retry or resample	Failures are transient or multiple independently graded candidates are useful.	Safety against duplicated side effects or shared systematic error.
Fall back	A narrower deterministic capability can provide useful service.	Pretending the fallback has the same scope or quality.
Escalate	Judgment, authority, or context must come from a responsible person or specialist.	Adding a ceremonial approval that reviewers cannot meaningfully assess.

Responses can be composed. A research system may retrieve, detect conflicting evidence, ask the user to narrow the time period, and then complete with a disclosed limitation. A purchasing agent may plan freely but re-read price and inventory, validate policy, and request approval immediately before commitment.

Evaluate a distribution, not a demonstration

For variable behavior, an evaluation case is not fully described by one input and one recorded output. It also needs the acceptance criteria, relevant environment, number of trials, and aggregation rule.

Anthropic’s agent-evaluation article describes two aggregations that answer different product questions (Anthropic, 2026):

pass@k asks whether at least one of k trials succeeds. It fits candidate generation when the trials are sampled under a defined setup and a reliable verifier can select a valid result.
pass^k asks whether all k trials succeed. It is a consistency measure for products that need repeated uses to succeed, not a replacement for the observed per-trial success and failure rates.

As k increases, pass@k is non-decreasing and pass^k is non-increasing by definition. This does not mean that running more trials makes the underlying model simultaneously better and worse; the metrics ask different questions. Calculating pass^k as a simple power of a one-trial success rate additionally assumes independent, identically distributed trials, an assumption that shared prompts, models, and environments can violate. Report the trial setup and observed results rather than relying on the formula alone.

Sampling ten plans can be useful if a verifier checks them and the system selects one before acting. Retrying a purchase operation ten times is dangerous if any trial can commit a side effect. Candidate generation and effectful retries are different system designs even if both involve repeated model calls.

Repeated trials should vary only what the test intends to study. Pin model, prompt, tools, data, and environment when measuring run-to-run variability. Deliberately vary them when testing robustness to change. Otherwise model regressions, harness defects, and environmental noise become indistinguishable.

Offline performance also does not settle deployment behavior. Research on underspecification shows that predictors with equivalent test performance on the training-domain distribution can behave differently in deployment domains (D’Amour et al., 2022). This result concerns trained predictors, not prompt-to-prompt randomness, but it supports a broader warning: one aggregate test score does not establish every behavior that matters in deployment. Production readiness therefore needs tests and monitoring around the whole system, a point made in earlier production-ML work as well (Breck et al., 2016).

Treat confidence as evidence, not authority

A model’s confident tone, a numerical self-rating, agreement across samples, and empirical calibration are different signals.

Verbal confidence is especially easy to misread. Human-subject research found that people can overestimate model accuracy when shown default explanations, and that longer explanations can increase confidence without increasing correctness (Steyvers et al., 2025). A polished rationale should not receive more authority merely because it sounds complete.

Some uncertainty methods are useful within a defined task. Semantic entropy compares the meanings of multiple generations and can detect a subset of confabulations (Farquhar et al., 2024). But agreement does not prove truth: all samples may share the same missing evidence or misconception. More generally, uncertainty estimates can degrade under dataset shift (Ovadia et al., 2019).

Use a confidence estimate only after answering:

What exactly does the signal estimate?
On which task distribution was it calibrated?
Does calibration hold for the current model, prompt, tools, and population?
What action follows each range of the signal?
Which independent checks remain necessary?

If those answers are absent, the number may decorate uncertainty rather than manage it.

Trade-offs

Choice	Benefit	Cost or risk
Narrow acceptance criteria	Easier verification and more consistent outcomes	Rejects some useful variation and can overfit evaluation examples
Broad acceptance criteria	Preserves creativity and adaptability	Harder grading, greater review burden, and more room for hidden defects
More clarification	Reduces ambiguity before execution	Interrupts flow and transfers effort to the user
More retrieval and verification	Improves grounding and catches some errors	Adds latency, cost, source risk, and new failure modes
More samples or retries	Can find a valid candidate or overcome transient failure	Increases cost and may amplify shared errors or side effects
More hard constraints in application code	Protects authorization, data integrity, and side effects	Reduces model discretion and increases application complexity
More human review	Adds judgment and accountability	Creates queues, fatigue, inconsistency, and scaling limits

There is no globally correct position. Acceptance should become stricter as consequences become larger or less reversible, uncertainty rises, and recovery becomes harder.

Failure modes

Fluent certainty

The interface presents a coherent answer without evidence, limitations, or a way to express unresolved uncertainty. Users mistake presentation quality for correctness.

One-run validation

A team approves a capability after a convincing demonstration. Rare failures, unstable routing, and inconsistent trajectories remain invisible.

String-equality reliability

Tests demand one reference sentence for a task with many valid answers. They punish useful diversity while missing unsupported claims that happen to resemble the reference.

Consensus as truth

Several samples agree, so the system treats the answer as verified. Shared training bias, missing evidence, or a leading prompt can make every sample wrong in the same way.

Hidden configuration drift

Model, prompt, retrieval index, tool, or policy changes without versioned evaluation. Behavioral changes are misclassified as random model noise.

Retry into a side effect

A timeout obscures whether an action committed. The system retries generation and execution together, producing duplicate messages, purchases, or writes.

Universal confidence threshold

One score controls extraction, research, creative writing, and high-impact actions despite different calibration and consequences.

Abstention without service

The system refuses but offers no missing evidence, next action, narrower fallback, or escalation path. Risk decreases, but usefulness collapses.

Fieldbook guidance

For every capability that uses a generative model, complete a review of uncertainty before optimizing prompts:

Name the decision or artifact. What can vary, and what consequence follows?
Ask the four questions. Which request ambiguity, evidence gap, run-to-run variation, and environmental change matter?
Define acceptance. Specify valid output, permitted process, successful outcome, and allowed variation.
Identify hard constraints. Move authorization, policy, schema, reconciliation, transaction, and side-effect checks outside model judgment.
Handle unresolved cases. Define clarification, insufficient evidence, conflict, approval, retry, fallback, and escalation paths.
Choose evidence. Record provenance, configuration, state snapshots, trajectories, outcomes, and human decisions needed for review.
Evaluate repeatedly. Select trials and aggregation rules that match how the product is actually used.
Test change. Re-evaluate model, prompt, retrieval, tool, policy, and environment updates as versioned system changes.

A compact review record can use this form:

Capability:
Allowed variation:
Unacceptable variation:
Input ambiguity:
Knowledge gaps:
Behavioral variability:
Environmental changes:
Hard constraints enforced by the application:
Unresolved states and routes:
Trial and grading method:
Evidence retained:

The test is simple: when two runs differ, can the team determine whether both are acceptable, why they differed, and what should happen next? If not, the system shows variable behavior but is not yet designed to handle it reliably.

Checklist

References and how they are used

NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1 (2024) — a voluntary, cross-sector risk-management profile. This chapter uses its definition and discussion of confabulation and its system-lifecycle framing; it is not an application architecture or proof that one mitigation works.
Anthropic, “Demystifying evals for AI agents” (2026) — Anthropic engineering article based on internal and customer work. It supports the use of multiple trials, outcome and trajectory grading, isolated environments, and the distinction between pass@k and pass^k. It is not a peer-reviewed statistical standard.
Kalai et al., “Why Language Models Hallucinate” (2025) and the OpenAI research summary — a theoretical account of why pretraining produces some factual errors and why accuracy-only evaluation rewards guessing. The chapter uses the evaluation-incentive argument; the paper does not establish a universal abstention policy for every product.
Farquhar et al., “Detecting hallucinations in large language models using semantic entropy” (2024) — peer-reviewed research showing that meaning-level disagreement across samples can identify a subset of arbitrary wrong generations called confabulations. It does not detect systematic errors and does not make agreement proof of truth.
Ovadia et al., “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift” (2019) — a NeurIPS study of predictive uncertainty for supervised classification under distribution shift. It supports rechecking calibration after deployment conditions change, but it does not evaluate free-form language generation or agent trajectories.
D’Amour et al., “Underspecification Presents Challenges for Credibility in Modern Machine Learning” (2022) — a JMLR study showing that predictors with equivalent in-domain test performance can differ in deployment. It concerns underspecified training pipelines, not stochastic decoding or prompt-time variability.
Steyvers et al., “What large language models know and what people think they know” (2025) — peer-reviewed human-subject research on multiple-choice and short-answer question answering. It supports the claims about default explanations, explanation length, and user confidence; generalization to long-form or agentic tasks remains untested in that study.
Breck et al., “What’s your ML test score? A rubric for ML production systems” (2016) — a NIPS workshop paper proposing actionable tests for production ML. The chapter uses its system-level testing and monitoring lesson, while recognizing that the rubric predates modern generative models and agents.