1. Why Route
Key Points
- Every request sits inside a three-way tension — quality, cost, and latency. A single model forces one fixed point in that triangle for all traffic.
- The core idea of routing: send each request to the cheapest model that can handle it, instead of paying frontier prices for every call.
- Tiers differ in price by roughly 15–60x; that spread is why a router can misroute some queries and still net large savings (real deployments report 40–85% bill reduction).
- Modern assistants define three tiers — small / medium / large (cheap / mid / strong) — by increasing reasoning depth and cost.
- RouteLLM cut cost >85% on MT Bench while retaining ~95% of frontier quality, using the frontier model for only ~14% of calls (about 1 in 7).
A hyper-personalized assistant answers thousands of different kinds of requests. "What's the weather?" and "Draft a legally-sound severance clause tailored to my employment history" arrive through the same text box, yet demand wildly different amounts of intelligence. Sending both to a frontier reasoning model is like dispatching a trauma surgeon to treat a paper cut: it works, but it is slow, expensive, and wasteful. A router inspects an incoming request, estimates how hard it is, and dispatches it to the appropriate model tier.
The triage nurse analogy. Walk into a busy ER and you first meet a triage nurse who, in seconds, assigns a severity level: a sprained ankle to a GP, chest pain to cardiology, a trauma to the surgical team. The nurse is cheap and fast and does not treat you — they route you. A model router is the triage nurse of your AI system.
Model tiers and capabilities
To route, you first need a menu of models arranged by capability and cost. The prices below are illustrative; what matters is the shape of the ladder.
| Tier | Illustrative model | Role | Relative cost | Latency | Best for |
|---|
| Triage / cheap | Haiku 4.5 | Fast first responder | ~1x (baseline) | Lowest | Classification, extraction, short factual answers, formatting, the router itself |
| Mid | Sonnet 5 | Workhorse | ~5–15x | Medium | Most conversational turns, standard tool use, summarization, drafting |
| Strong / reasoning | Opus 4.8 | Specialist | ~40–60x | Highest | Multi-step reasoning, hard math/code, ambiguous or high-stakes requests, complex planning |
When one model is not enough. Routing earns its complexity where a single fixed model fails: bimodal traffic (a flood of trivial requests interleaved with rare critical ones), cost pressure at scale (the 15–60x spread makes "always frontier" a budget-destroyer), and latency-sensitive surfaces (autocomplete needs sub-second responses only a small model delivers).
[Animation slot: Cost/quality/latency triangle — a draggable point sliding between the three vertices, showing how a single-model choice locks one position while routing distributes requests across the whole space.]
Key Takeaway
- Routing exists to escape the single-model compromise. Because tiers differ in price by 15–60x, sending each request to the cheapest capable model yields 40–85% cost savings even when the router is imperfect — often while retaining ~95% of frontier quality.
2. Routing Strategies
Key Points
- Three broad families, simplest to most sophisticated: rule-based, learned (classifier / LLM-based), and cascades. They are not mutually exclusive.
- Rule-based (keyword) routing is deterministic, transparent, and near-zero-latency, but brittle to phrasing and easily gamed — best used as a cheap floor and safety guardrail.
- Learned routing treats routing as a classification task; RouteLLM trains four router types and generalizes to new model pairs without retraining.
- The single tunable dial in RouteLLM is the cost threshold: higher threshold = lower cost but possibly lower quality.
- A router decides up front and pays once; a cascade runs cheap first, scores the answer, and escalates only if confidence is low — paying twice only on escalation.
Rule-based routing
The simplest router is a set of deterministic keyword rules: words like "sum," "list," "define" indicate low complexity; words like "prove," "derive," "explain why" suggest high complexity. Rules are an excellent floor — they cheaply catch obvious cases and hand everything ambiguous to a safe default — and invaluable as guardrails (e.g., "always route legal or medical content to the strong tier, regardless of the classifier").
Figure 12.1: Router decision flow — classify difficulty up front, then dispatch to a single tier.
flowchart TD
A[Incoming request] --> B{"Attached files or<br/>long context?"}
B -->|yes| S["Strong tier<br/>(Opus 4.8)"]
B -->|no| C{"Hard-complexity<br/>cue matched?"}
C -->|yes| S
C -->|no| D{"Easy-complexity<br/>cue matched?"}
D -->|yes| E["Cheap tier<br/>(Haiku 4.5)"]
D -->|no| F["Mid tier<br/>(Sonnet 5)<br/>safe default"]
Classifier and LLM-based routing
To capture meaning beyond surface tokens, a learned router reframes routing as a classification task. Two common approaches: a lightweight ML classifier (e.g., DistilBERT trained on ~31,019 prompts from eight benchmarks) that captures semantics, or LLM-based routing where a cheap LLM judges difficulty directly. The reference framework, RouteLLM, focuses on routing between a stronger/expensive and a weaker/cheaper model, training four router types:
| Router type | How it decides |
|---|
| Similarity-weighted (SW) ranking | Weighted Elo based on similarity between the incoming prompt and labeled examples |
| Matrix factorization | Learns a scoring function for model–prompt compatibility (best on MT Bench) |
| BERT classifier | Predicts which model will produce the superior response |
| Causal LLM classifier | A generative classifier that also predicts which model performs better |
Crucially, RouteLLM routers generalize to new model pairs without retraining — you can swap in a new mid-tier model without re-collecting preference data. The one tunable knob is the cost threshold, which slides an operator along the cost-quality frontier.
Model cascades and escalation
A cascade inverts the logic: it runs the cheapest model first, checks a confidence/quality signal, and escalates to a stronger model only if the answer is not good enough. The foundational work is FrugalGPT, which reports matching the best individual model's accuracy with up to 98% cost reduction, combining Prompt Adaptation, LLM Approximation (caching), and the LLM Cascade.
Figure 12.2: Model cascade with escalation — run cheapest first, score, escalate only if needed.
flowchart TD
A[Incoming request] --> B["Query cheap model<br/>(Haiku 4.5)"]
B --> C{"Confidence above<br/>threshold?"}
C -->|yes| R["Return answer"]
C -->|no| D["Escalate to mid model<br/>(Sonnet 5)"]
D --> E{"Confidence above<br/>threshold?"}
E -->|yes| R
E -->|no| F["Escalate to strong model<br/>(Opus 4.8)"]
F --> R
Router vs. cascade — the core tradeoff. This is the single most important design decision in the chapter.
| Dimension | Router (decide up front) | Cascade (run, then escalate) |
|---|
| When decision is made | Before any model runs | After observing a real cheap-model answer |
| Information available | Must predict difficulty blind | Sees an actual answer to judge |
| Cost on easy requests | Pay once (chosen model) | Pay once (cheap model only) |
| Cost on hard requests | Pay once (strong model) | Pay twice — cheap plus strong |
| Latency on escalation | None extra | Added tail latency (waits for both) |
| Decision quality | Only as good as difficulty prediction | Grounded in a real, observed answer |
| Best when | Prediction is reliable; latency matters | Cheap model is often right; a good quality signal exists |
[Animation slot: Side-by-side router vs. cascade — a single request flows through each path in parallel, animating the "pay once" arrow for the router and the "pay twice + wait" double-hop for the cascade on a hard request.]
Key Takeaway
- Rule-based routing is transparent and fast but brittle; learned routing (RouteLLM's four types) captures semantic difficulty and generalizes across model pairs. A router decides blind and pays once; a cascade (FrugalGPT) observes a real answer and escalates only when confidence is low, paying twice only on escalation.
3. Triage in Practice
Key Points
- Complexity classification runs in three modes: keyword-based (fast, brittle), ML-based (semantic, needs a labeled set), and hybrid (rules first, classifier for the fuzzy middle) — the hybrid is usually the production sweet spot.
- Routers draw on difficulty signals beyond keywords: semantic embeddings, predicted-quality scoring, and uncertainty signals (both a query's complexity and the model's uncertainty).
- In a cascade, everything hinges on the confidence threshold: too permissive leaks errors; too strict over-escalates and savings evaporate.
- Most deployed confidence scores are uncalibrated; tune thresholds on your own traffic. Calibration-first methods like UCCI (isotonic regression) exist precisely for this.
- Route on the kind of difficulty: reasoning-heavy tasks need the strong tier; tool-heavy tasks can often stay cheap when routing and tools are reliable.
Detecting hard requests
Consider the request: "Given my three savings accounts and last year's tax return, tell me how much I can contribute to my IRA without penalty." The rule layer finds no easy keyword and flags the attached files; the classifier layer embeds the prompt, finds it close to labeled "multi-constraint reasoning" examples, and predicts complexity = high; the decision routes to the strong tier. Contrast "What's the contribution limit for an IRA this year?" — a bare lookup the cheap tier answers instantly.
Confidence and escalation thresholds
Get the threshold wrong in either direction and you lose:
- Too permissive (accepts low-confidence answers) → the cascade leaks errors: bad cheap-model answers reach users.
- Too strict (escalates readily) → the cascade over-escalates: nearly everything goes to the strong model and savings evaporate.
A sobering caution: most deployed routers and cascades use uncalibrated confidence scores and require per-workload tuning. Research like UCCI maps token-level margin uncertainty to a per-query error probability via isotonic regression, then picks the threshold by constrained cost minimization. The practical recipe: never ship a hand-guessed threshold — collect labeled samples of your own traffic, plot escalation and error rates against the threshold, and re-check periodically as traffic drifts.
Tool-heavy vs. reasoning-heavy
Reasoning-heavy tasks demand deep multi-step thinking within the model (proofs, hard math/code, ambiguous planning) — the classic case for the strong reasoning tier. Tool-heavy tasks are hard because they need many correct tool calls; a capable cheap model that reliably picks the right tool often beats an expensive one because the tools supply the correctness. The heuristic: route on the source of difficulty, not just its magnitude.
[Animation slot: A threshold slider sweeping from permissive to strict, with two live counters — "errors leaked" rising on the left and "over-escalation cost" rising on the right — showing the sweet spot in the middle.]
Key Takeaway
- Detect hard requests with a hybrid of keyword rules, a learned complexity classifier, and uncertainty signals. Tune escalation thresholds on your own traffic (scores are usually uncalibrated; UCCI-style isotonic regression calibrates them). Route on the kind of difficulty: reasoning-heavy to the strong tier, tool-heavy often staying cheap.
4. Operational Concerns
Key Points
- A fallback handles availability (intended model unreachable, rate-limited, timing out); an escalation handles quality. Every production router needs within-provider, cross-provider, and timeout-budget fallbacks.
- Fallbacks should degrade quality gracefully, never fail silently — log every fallback event so you know when "healthy" traffic is running on a downgraded model.
- Measure continuously: cost per request (target 40–85% savings), quality retention (~95% north star), tier distribution, escalation/fallback rates, and latency percentiles (especially the tail).
- The signature failure is a silent quality regression: real cost savings mask the harm because routing pays off even with imperfect classification.
- Guardrails: strong-tier floors for high-stakes categories, shadow/canary rollouts, an unconditional control group, and scheduled recalibration on drift.
Fallbacks and provider outages
Escalation is about quality; a fallback is about availability — you move to an alternative model (possibly a different provider) because your intended model is unreachable. Every production router needs a fallback chain: within-provider (if Opus is rate-limited, retry Sonnet rather than error out), cross-provider (keep a comparable model from a second vendor warm), and timeout budgets per tier. The principle: degrade quality gracefully, never fail silently — and log every fallback event. FrugalGPT's caching doubles as an availability aid, serving cached responses even when a provider is down.
Measuring routing quality
You cannot manage what you do not measure, and a router is deceptively easy to think is working while it quietly degrades. Track at minimum:
- Cost per request and overall bill reduction (target band 40–85%).
- Quality retention (~95% of frontier quality) via an LLM judge, human preference sampling, or task-specific correctness checks.
- Tier distribution — RouteLLM's best MT-Bench router used only 14% frontier calls; sudden drift signals a problem.
- Escalation rate (cascades) and fallback rate (availability) — spikes are early warnings.
- Latency percentiles, especially p95/p99 — users feel the tail, not the average.
Avoiding quality regressions
The most dangerous failure mode is a silent quality regression: costs look great, dashboards are green, but a segment of users now gets worse answers. Because the price spread is so forgiving, savings can mask the harm. Guardrails:
- Set a floor. Hard-code rules pinning high-stakes categories (legal, medical, financial, safety) to the strong tier regardless of the classifier.
- Shadow and canary. Run any router/threshold change on a small slice first, comparing quality before rollout.
- Hold out a control. Route a small random fraction of all traffic to the strong tier unconditionally for a ground-truth comparison.
- Recalibrate on drift. Re-run threshold tuning on fresh data on a schedule, and after any model swap.
- Watch the tradeoff dial. A higher cost threshold means lower cost but potentially lower quality — treat it as a product decision.
[Animation slot: A monitoring dashboard where the "cost saved" gauge climbs green while a hidden "quality retention" line quietly dips — then the control-group comparison flags the divergence, illustrating a silent regression being caught.]
Key Takeaway
- Fallbacks handle availability (degrade gracefully, never fail silently); escalation handles quality. Measure cost per request, quality retention (~95% target), tier distribution, escalation/fallback rates, and tail latency continuously. Defend against silent quality regressions with strong-tier floors, canary rollouts, an unconditional control group, and scheduled recalibration.