Model Strategy and Routing

An agentic system rarely has one universally best model. A model that is suitable for classification may be a poor choice for long-horizon tool use; a model that handles the hardest case may be unnecessarily slow or expensive for routine steps. Providers, model versions, modalities, context limits, deployment locations, and safety characteristics also change.

The architectural problem is therefore not “Which model is best?” It is: which model-mediated decision should be made by which eligible configuration, under which evidence and fallback policy?

Thesis

Model routing is a versioned decision policy over eligible model configurations. It must preserve task quality and risk constraints before optimizing latency or cost.

A configuration includes more than a model name. It may include a provider and version, reasoning or inference settings, tools, response schema, prompt or adapter version, and retry or fallback behavior. Changing any of these can change system behavior and should be evaluated as a system change.

Start with decision classes, not one product-wide model

Route the smallest unit whose requirements are reasonably stable. One run may contain several decision classes:

Decision class	Typical requirement
Intent or document classification	Stable labels, low latency, calibrated abstention
Evidence synthesis	Source use, long context, citation fidelity
Tool selection and arguments	Schema adherence, instruction following, tool-use reliability
Planning	Constraint handling, decomposition, revision from feedback
High-consequence recommendation	Stronger evidence, conservative uncertainty handling, independent checks
User-facing explanation	Language quality, audience fit, faithful representation of results

Assigning one model to the whole “agent” hides these differences. It also makes upgrades risky: a new model can improve synthesis while regressing tool arguments or multilingual behavior. Evaluation should therefore report quality by decision class and end-to-end outcome.

Eligibility comes before optimization

Before comparing quality, cost, or speed, remove configurations that cannot responsibly serve the request. Eligibility can depend on:

• required modality, context capacity, structured output, or tool support;
• data residency, retention, confidentiality, and tenant policy;
• language and domain capability demonstrated on relevant evaluations;
• maximum latency, availability, and deployment constraints;
• permission to expose particular data or tools to that provider; and
• risk rules that require a named model class, verifier, or human approval.

This order matters. A cheap model is not a valid route when it cannot receive the data, express the required action contract, or meet a mandatory quality threshold. The eligibility decision should be enforceable outside the model and recorded with the route.

Separate selection, adaptation, and routing

These choices are related but distinct:

• Model selection chooses the base model families and versions a system supports.
• Adaptation changes how a model behaves through instructions, examples, retrieval, tools, structured outputs, fine-tuning, or other provider-supported controls.
• Routing chooses among eligible configurations for a particular decision.
• Escalation spends additional computation or moves to a stronger configuration after a weak result, failed check, or uncertain outcome.

Do not fine-tune by default to repair missing context, a weak tool contract, or an incorrect control loop. First identify the actual failure. Adaptation is justified when maintained evaluation data shows that a repeatable behavior gap is better addressed there than in context, tools, deterministic code, or product scope.

Bind context configuration to the model

The evaluated candidate is not the model alone. It is the model paired with its instructions, examples, tool definitions, schemas, Skill triggers, context selection rules, and inference settings. A prompt that is useful scaffolding for one model can be redundant or constraining for another. Conversely, a lean configuration that succeeds with a newer model can regress on an older, smaller, or differently trained model.

Start from a shared configuration when it works. Fork a model-specific variant only when representative evaluation justifies the additional review, caching, release, and observability surface. Record which requirement or measured gap each difference serves so variants do not become unowned branches.

Treat an upgrade as a two-way context review. Test what the new model breaks, but also whether inherited instructions, examples, and tools remain necessary. Establish the exact working configuration, change one bounded group, and rerun the same tasks and trials. Anthropic reports that newer Claude Code configurations supported substantially leaner prompts than earlier models, and OpenAI recommends the same incremental, evaluation-driven simplification process (Anthropic, 2026; OpenAI, current guidance). Both are vendor-internal coding-agent evidence, so neither defines the correct configuration for another workload.

The same review applies to Skill descriptions, repository guidance, and task prompts: they are model-facing configuration, not neutral documentation. A dated OpenAI report on GPT-6 Astra recommends auditing whether these layers still earn their cost as model capability changes. Treat that as a trigger for paired regression and bounded-removal tests, not as evidence that any model can safely work without explicit context (OpenAI, 2026).

Routing patterns

Static assignment

Map each decision class to one configuration. This is often the best starting point: it is understandable, easy to evaluate, and operationally predictable. Its weakness is that every request in a class receives the same capability and cost.

Deterministic rules

Use observable properties—modality, language, document size, tenant policy, action risk, or a user-selected mode—to select a route. Rules work well when requirements are explicit. They become brittle when “difficulty” is inferred from superficial proxies.

Learned pre-generation routing

A classifier or learned router predicts which model can handle the request before generation. RouteLLM demonstrates that routers learned from preference data can reduce cost on its evaluated benchmark while preserving a chosen quality level (Ong et al., 2025). That result supports the feasibility of learned routing, not a universal saving: performance depends on the candidate models, traffic distribution, labels, and threshold.

Cascades

Run a lower-cost configuration first, then escalate when its result fails a confidence, verifier, or quality gate. FrugalGPT studied cascades that combine model calls to trade accuracy against cost (Chen et al., 2024). Cascades can save computation on easy cases, but the gate becomes the critical component. If it cannot detect convincing errors, the strongest model is never reached.

Parallel candidates and selection

Generate multiple candidates and select or synthesize them. This can help when diversity is valuable and a reliable evaluator exists. Correlated errors, extra latency, and a weak selector can make the pattern worse than a single strong call. Agreement between similar models is not independent proof.

These patterns may be combined, but every additional branch expands the behavior and evaluation surface.

Treat confidence as a measurement problem

A model’s self-reported confidence is not a provider-independent probability of correctness. Token probabilities, verbal confidence, consistency across samples, router scores, and verifier scores measure different things. They can be useful only after calibration against representative outcomes.

Define the routing gate in operational terms:

eligible configurations
task and risk features
selected route and reason code
quality or completion check
escalation condition
fallback behavior when dependencies fail
maximum calls, latency, and cost
evidence retained for evaluation

Prefer gates tied to observable task conditions: schema validity, required evidence present, tests passing, policy constraints satisfied, or an evaluator whose error rates are known. A polished answer or high self-confidence should not bypass a failed deterministic check.

Fallback is not automatically safe

A fallback changes capability and sometimes data handling, tool support, or output semantics. Define whether each failure permits:

• retrying the same configuration for a transient provider error;
• using an equivalent deployment of the same model;
• degrading to a weaker model with a narrower promise;
• escalating to a stronger model;
• switching provider after applying its eligibility rules; or
• returning a partial, waiting, blocked, or failed result.

Do not silently send sensitive context to a new provider or let a fallback model perform an action it has not been evaluated to propose. A graceful fallback may be “save the run and ask the user to resume later,” not “answer at any quality.”

Evaluation and change management

Routing should be evaluated at three levels:

1. Per-decision quality: Did each routed call satisfy its contract?
2. End-to-end outcome: Did the system reach the correct task state under its control loop?
3. Policy performance: How often did the router choose each path, escalate, fail, or violate latency and cost limits?

Report conditional results, not only averages. A router can reduce mean cost while concentrating errors in a language, tenant, rare action, or high-risk class. Compare it with simple baselines: one strong model, one cheaper model, and a deterministic rule. Include provider failures and model-version changes in tests.

Evaluate the exact context configuration paired with each candidate. A model-level comparison that silently reuses, adds, or removes different scaffolding cannot attribute the behavior change. When a candidate is upgraded, run both regression tests and bounded removal tests for inherited context.

Pin versions when the provider supports it; otherwise record the actual served model and rerun release gates when behavior changes. Shadow routing can compare a proposed policy without allowing it to affect users. Production Engineering will own rollout, capacity, and incident practice; Evaluation owns the general measurement method. This chapter owns the architectural fact that the route itself is observable, versioned behavior.

Failure modes

“Hard query” as the entire policy

A vague difficulty score ignores privacy, tools, modality, risk, and task-specific quality. Define eligibility and decision-class requirements first.

Cheapest-first without a reliable gate

The system accepts fluent errors because the cascade has no independent signal for escalation. Measure gate false accepts, not only escalation rate.

Product-wide routing

One early classification assigns the same model to planning, actions, verification, and explanation. Route by material decision class where the requirements differ.

Hidden fallback

A provider failure silently changes models or providers. Users and operators cannot explain the behavior, and policy constraints may be broken. Record the route and preserve the promise or fail explicitly.

Benchmark inheritance

A public ranking is treated as proof for a local workflow. Benchmarks inform candidates; application evaluations decide eligibility.

Router overhead ignored

The classification call, duplicate context transfer, cache fragmentation, and escalation latency erase expected savings. Measure the whole path.

Model name as configuration

Prompts, tools, schemas, inference settings, and safety settings drift while dashboards group them under one model label. Version the complete behavior configuration.

Universal prompt across models

Every candidate inherits the same prompt because it is operationally convenient, even when one model needs additional scaffolding or another is constrained by it. Keep a shared baseline where possible, but evaluate the exact model-context pair and fork only for a measured reason.

Fieldbook guidance

Maintain a route card for every material decision class:

Field	Question
Decision	What bounded judgment or artifact is being produced?
Eligibility	Which modality, data, policy, tool, and risk constraints apply?
Candidates	Which versioned model-plus-context configurations passed relevant evaluation?
Policy	Static assignment, rule, learned router, cascade, or parallel selection?
Gate	What measured condition accepts, escalates, abstains, or fails?
Fallback	Which failures permit which alternate route without changing the promise?
Limits	What bounds calls, latency, tokens, and cost?
Record	Which features, route, versions, reason, results, and checks are retained?
Review	What distribution shift, model change, or context change triggers regression and removal tests?

Checklist

• Are routes defined for bounded decision classes rather than a vague whole-agent difficulty?
• Are data, modality, tool, policy, and risk constraints enforced before optimization?
• Is each candidate a versioned configuration, not only a model name?
• Are instructions, examples, tools, schemas, triggers, and context rules evaluated with the model they serve?
• Are model-specific variants limited to measured needs with an owner and removal condition?
• Has the simplest static route been measured as a baseline?
• Are learned scores and model confidence calibrated on representative outcomes?
• Does a cascade gate detect plausible but wrong results?
• Are fallback paths subject to the same eligibility and authority rules?
• Can a run report the selected route, reason, model, configuration, and escalation history?
• Do evaluations cover decision quality, end-to-end outcomes, and policy performance by subgroup?
• Do model or provider changes trigger controlled reevaluation and rollout?

References and how they are used

• Ong et al., “RouteLLM: Learning to Route LLMs with Preference Data” — peer-reviewed evidence that learned routing can trade model cost against quality on evaluated tasks; its reported savings are not generalized beyond those conditions.
• Chen et al., “FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance” — peer-reviewed study of model selection and cascades used to explain staged escalation and its dependency on quality estimation.
• Ding et al., “Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing” — research evidence for query-level routing; used as an additional design example, not as a production guarantee.
• OpenAI, “A practical guide to building agents” — OpenAI product-adjacent agent-building article used for the practical strategy of establishing an evaluation baseline with a capable model before replacing eligible steps with smaller models.
• Anthropic, “The new rules of context engineering for Claude 5 generation models” (2026) — Anthropic engineering report used as a dated example that prompt and example needs can differ by model generation. Its internal Claude Code evaluations do not establish how other models should be configured.
• OpenAI, “Model guidance: Favor leaner prompts” — current OpenAI developer guidance supporting one-change-at-a-time prompt simplification and retention of examples that encode requirements or repair measured gaps.
• OpenAI, “Rethinking skills and prompts for GPT-6 Astra” — dated OpenAI engineering evidence used to treat Skill descriptions, repository guidance, and task prompts as part of a model-specific configuration and to trigger regression plus bounded-removal tests after upgrades. It provides no cross-model benchmark or universal scaffolding rule.