epochs

Learn · Module 13 · Production AI and MLOps · Lesson 1 of 7

Model routing

Not every request needs your most expensive model. Routing sends each one to the cheapest model that can handle it.

3 minute read

In short
  • A router sends each request to the cheapest model that can handle it well.
  • Sending 70% of traffic to a small model can cut costs by more than half.
  • Cascades need a reliable check to know when to escalate.

In plain words

A router looks at a request and decides which model should answer it. Simple, common requests go to a small, fast, cheap model. Hard or high-stakes ones go to a large model. Done well, most traffic gets cheaper and faster with no drop in quality.

Worked example: what routing saves

From the cost lesson: a request costs $0.0105 on the large model and $0.000875 on the small one. Suppose 70% of requests are simple enough for the small model.

  1. 1
    Blended cost per request
    0.7 × $0.000875 + 0.3 × $0.0105 = $0.00376
  2. 2
    At 50,000 requests a day
    $188 a day, instead of $525
  3. 3
    Saving
    64% cheaper

Ways to route

methodhow it decidestrade-off
RulesRequest type, length, user planSimple and predictable; misses subtle cases
ClassifierA small model labels each request easy or hardSmarter; needs labelled examples
CascadeTry the small model; escalate if it's unsure or fails a checkGood quality; slower when escalating
Your turn

In a cascade, the small model answers first. What's the key requirement for this to work?

What to remember

Measure quality per route, not just overall, so you notice if the small model is failing a particular kind of request. Start with simple rules, and only add a learned router when the data shows it's worth it.

Common mistakes

  • Routing by guesswork instead of measuring quality per route on an evaluation set.
  • Using an expensive model to decide which model to use, which eats the savings.
  • Never re-checking routes when models or prices change.
Check yourself

Answer in your own words, out loud or on paper, before you open the model answer.

How would you reduce LLM costs without hurting quality?

First I'd measure where the money goes: which features, which prompts, how many tokens. Then the main levers. Route requests by difficulty, sending simple ones, like classification or short factual answers, to a small cheap model and only hard ones to the large model, using a lightweight classifier or rules, and check quality per route on an evaluation set. Cache shared prompt prefixes and repeated answers. Trim prompts and cap output length. Batch work that isn't urgent, since batch APIs are often cheaper. For a narrow, high-volume task, consider fine-tuning a small model. After each change, I'd confirm on the evaluation set that quality held, and monitor it in production.

More to think about
  • Design a router between a small and a large model.
  • How would you know if routing is hurting quality?

Go further

Was this lesson clear?