Model Strategy and Routing

An agentic system rarely has one universally best model. A model that is suitable for classification may be a poor choice for long-horizon tool use; a model that handles the hardest case may be unnecessarily slow or expensive for routine steps. Providers, model versions, modalities, context limits, deployment locations, and safety characteristics also change.

The architectural problem is therefore not “Which model is best?” It is: which model-mediated decision should be made by which eligible configuration, under which evidence and fallback policy?

Thesis

Model routing is a versioned decision policy over eligible model configurations. It must preserve task quality and risk constraints before optimizing latency or cost.

A configuration includes more than a model name. It may include a provider and version, reasoning or inference settings, tools, response schema, prompt or adapter version, and retry or fallback behavior. Changing any of these can change system behavior and should be evaluated as a system change.

Start with decision classes, not one product-wide model

Route the smallest unit whose requirements are reasonably stable. One run may contain several decision classes:

Decision class	Typical requirement
Intent or document classification	Stable labels, low latency, calibrated abstention
Evidence synthesis	Source use, long context, citation fidelity
Tool selection and arguments	Schema adherence, instruction following, tool-use reliability
Planning	Constraint handling, decomposition, revision from feedback
High-consequence recommendation	Stronger evidence, conservative uncertainty handling, independent checks
User-facing explanation	Language quality, audience fit, faithful representation of results

Assigning one model to the whole “agent” hides these differences. It also makes upgrades risky: a new model can improve synthesis while regressing tool arguments or multilingual behavior. Evaluation should therefore report quality by decision class and end-to-end outcome.

Eligibility comes before optimization

Before comparing quality, cost, or speed, remove configurations that cannot responsibly serve the request. Eligibility can depend on:

• required modality, context capacity, structured output, or tool support;
• data residency, retention, confidentiality, and tenant policy;
• language and domain capability demonstrated on relevant evaluations;
• maximum latency, availability, and deployment constraints;
• permission to expose particular data or tools to that provider; and
• risk rules that require a named model class, verifier, or human approval.

This order matters. A cheap model is not a valid route when it cannot receive the data, express the required action contract, or meet a mandatory quality threshold. The eligibility decision should be enforceable outside the model and recorded with the route.

Separate selection, adaptation, and routing

These choices are related but distinct:

• Model selection chooses the base model families and versions a system supports.
• Adaptation changes how a model behaves through instructions, examples, retrieval, tools, structured outputs, fine-tuning, or other provider-supported controls.
• Routing chooses among eligible configurations for a particular decision.
• Escalation spends additional computation or moves to a stronger configuration after a weak result, failed check, or uncertain outcome.

Do not fine-tune by default to repair missing context, a weak tool contract, or an incorrect control loop. First identify the actual failure. Adaptation is justified when maintained evaluation data shows that a repeatable behavior gap is better addressed there than in context, tools, deterministic code, or product scope.

Routing patterns

Static assignment

Map each decision class to one configuration. This is often the best starting point: it is understandable, easy to evaluate, and operationally predictable. Its weakness is that every request in a class receives the same capability and cost.

Deterministic rules

Use observable properties—modality, language, document size, tenant policy, action risk, or a user-selected mode—to select a route. Rules work well when requirements are explicit. They become brittle when “difficulty” is inferred from superficial proxies.

Learned pre-generation routing

A classifier or learned router predicts which model can handle the request before generation. RouteLLM demonstrates that routers learned from preference data can reduce cost on its evaluated benchmark while preserving a chosen quality level (Ong et al., 2025). That result supports the feasibility of learned routing, not a universal saving: performance depends on the candidate models, traffic distribution, labels, and threshold.

Cascades

Run a lower-cost configuration first, then escalate when its result fails a confidence, verifier, or quality gate. FrugalGPT studied cascades that combine model calls to trade accuracy against cost (Chen et al., 2024). Cascades can save computation on easy cases, but the gate becomes the critical component. If it cannot detect convincing errors, the strongest model is never reached.

Parallel candidates and selection

Generate multiple candidates and select or synthesize them. This can help when diversity is valuable and a reliable evaluator exists. Correlated errors, extra latency, and a weak selector can make the pattern worse than a single strong call. Agreement between similar models is not independent proof.

These patterns may be combined, but every additional branch expands the behavior and evaluation surface.

Treat confidence as a measurement problem

A model’s self-reported confidence is not a provider-independent probability of correctness. Token probabilities, verbal confidence, consistency across samples, router scores, and verifier scores measure different things. They can be useful only after calibration against representative outcomes.

Define the routing gate in operational terms:

eligible configurations
task and risk features
selected route and reason code
quality or completion check
escalation condition
fallback behavior when dependencies fail
maximum calls, latency, and cost
evidence retained for evaluation

Prefer gates tied to observable task conditions: schema validity, required evidence present, tests passing, policy constraints satisfied, or an evaluator whose error rates are known. A polished answer or high self-confidence should not bypass a failed deterministic check.

Fallback is not automatically safe

A fallback changes capability and sometimes data handling, tool support, or output semantics. Define whether each failure permits:

• retrying the same configuration for a transient provider error;
• using an equivalent deployment of the same model;
• degrading to a weaker model with a narrower promise;
• escalating to a stronger model;
• switching provider after applying its eligibility rules; or
• returning a partial, waiting, blocked, or failed result.

Do not silently send sensitive context to a new provider or let a fallback model perform an action it has not been evaluated to propose. A graceful fallback may be “save the run and ask the user to resume later,” not “answer at any quality.”

Evaluation and change management

Routing should be evaluated at three levels:

1. Per-decision quality: Did each routed call satisfy its contract?
2. End-to-end outcome: Did the system reach the correct task state under its control loop?
3. Policy performance: How often did the router choose each path, escalate, fail, or violate latency and cost limits?

Report conditional results, not only averages. A router can reduce mean cost while concentrating errors in a language, tenant, rare action, or high-risk class. Compare it with simple baselines: one strong model, one cheaper model, and a deterministic rule. Include provider failures and model-version changes in tests.

Pin versions when the provider supports it; otherwise record the actual served model and rerun release gates when behavior changes. Shadow routing can compare a proposed policy without allowing it to affect users. Production Engineering will own rollout, capacity, and incident practice; Evaluation owns the general measurement method. This chapter owns the architectural fact that the route itself is observable, versioned behavior.

Failure modes

“Hard query” as the entire policy

A vague difficulty score ignores privacy, tools, modality, risk, and task-specific quality. Define eligibility and decision-class requirements first.

Cheapest-first without a reliable gate

The system accepts fluent errors because the cascade has no independent signal for escalation. Measure gate false accepts, not only escalation rate.

Product-wide routing

One early classification assigns the same model to planning, actions, verification, and explanation. Route by material decision class where the requirements differ.

Hidden fallback

A provider failure silently changes models or providers. Users and operators cannot explain the behavior, and policy constraints may be broken. Record the route and preserve the promise or fail explicitly.

Benchmark inheritance

A public ranking is treated as proof for a local workflow. Benchmarks inform candidates; application evaluations decide eligibility.

Router overhead ignored

The classification call, duplicate context transfer, cache fragmentation, and escalation latency erase expected savings. Measure the whole path.

Model name as configuration

Prompts, tools, schemas, inference settings, and safety settings drift while dashboards group them under one model label. Version the complete behavior configuration.

Fieldbook guidance

Maintain a route card for every material decision class:

Field	Question
Decision	What bounded judgment or artifact is being produced?
Eligibility	Which modality, data, policy, tool, and risk constraints apply?
Candidates	Which versioned configurations passed relevant evaluation?
Policy	Static assignment, rule, learned router, cascade, or parallel selection?
Gate	What measured condition accepts, escalates, abstains, or fails?
Fallback	Which failures permit which alternate route without changing the promise?
Limits	What bounds calls, latency, tokens, and cost?
Record	Which features, route, versions, reason, results, and checks are retained?
Review	What distribution shift or model change triggers reevaluation?

Checklist

• Are routes defined for bounded decision classes rather than a vague whole-agent difficulty?
• Are data, modality, tool, policy, and risk constraints enforced before optimization?
• Is each candidate a versioned configuration, not only a model name?
• Has the simplest static route been measured as a baseline?
• Are learned scores and model confidence calibrated on representative outcomes?
• Does a cascade gate detect plausible but wrong results?
• Are fallback paths subject to the same eligibility and authority rules?
• Can a run report the selected route, reason, model, configuration, and escalation history?
• Do evaluations cover decision quality, end-to-end outcomes, and policy performance by subgroup?
• Do model or provider changes trigger controlled reevaluation and rollout?

References and how they are used

• Ong et al., “RouteLLM: Learning to Route LLMs with Preference Data” — peer-reviewed evidence that learned routing can trade model cost against quality on evaluated tasks; its reported savings are not generalized beyond those conditions.
• Chen et al., “FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance” — peer-reviewed study of model selection and cascades used to explain staged escalation and its dependency on quality estimation.
• Ding et al., “Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing” — research evidence for query-level routing; used as an additional design example, not as a production guarantee.
• OpenAI, “A practical guide to building agents” — OpenAI product-adjacent agent-building article used for the practical strategy of establishing an evaluation baseline with a capable model before replacing eligible steps with smaller models.