Learn · Module 13 · Production AI and MLOps · Lesson 1 of 7
Model routing
Not every request needs your most expensive model. Routing sends each one to the cheapest model that can handle it.
- A router sends each request to the cheapest model that can handle it well.
- Sending 70% of traffic to a small model can cut costs by more than half.
- Cascades need a reliable check to know when to escalate.
In plain words
A router looks at a request and decides which model should answer it. Simple, common requests go to a small, fast, cheap model. Hard or high-stakes ones go to a large model. Done well, most traffic gets cheaper and faster with no drop in quality.
Worked example: what routing saves
From the cost lesson: a request costs $0.0105 on the large model and $0.000875 on the small one. Suppose 70% of requests are simple enough for the small model.
- 1Blended cost per request
0.7 × $0.000875 + 0.3 × $0.0105 = $0.00376
- 2At 50,000 requests a day
$188 a day, instead of $525
- 3Saving
64% cheaper
Ways to route
| method | how it decides | trade-off |
|---|---|---|
| Rules | Request type, length, user plan | Simple and predictable; misses subtle cases |
| Classifier | A small model labels each request easy or hard | Smarter; needs labelled examples |
| Cascade | Try the small model; escalate if it's unsure or fails a check | Good quality; slower when escalating |
In a cascade, the small model answers first. What's the key requirement for this to work?
What to remember
Measure quality per route, not just overall, so you notice if the small model is failing a particular kind of request. Start with simple rules, and only add a learned router when the data shows it's worth it.
Common mistakes
- Routing by guesswork instead of measuring quality per route on an evaluation set.
- Using an expensive model to decide which model to use, which eats the savings.
- Never re-checking routes when models or prices change.
Answer in your own words, out loud or on paper, before you open the model answer.
How would you reduce LLM costs without hurting quality?
First I'd measure where the money goes: which features, which prompts, how many tokens. Then the main levers. Route requests by difficulty, sending simple ones, like classification or short factual answers, to a small cheap model and only hard ones to the large model, using a lightweight classifier or rules, and check quality per route on an evaluation set. Cache shared prompt prefixes and repeated answers. Trim prompts and cap output length. Batch work that isn't urgent, since batch APIs are often cheaper. For a narrow, high-volume task, consider fine-tuning a small model. After each change, I'd confirm on the evaluation set that quality held, and monitor it in production.
- Design a router between a small and a large model.
- How would you know if routing is hurting quality?
Go further
- PaperRouteLLM: Learning to Route LLMs with Preference DataOng et al., 2024
- PaperFrugalGPT: How to Use LLMs While Reducing CostChen, Zaharia & Zou, 2023