epochs

Learn · Module 13 · Production AI and MLOps · Lesson 2 of 7

Fallbacks

Every AI provider has outages, rate limits and slow days. Plan for them before they happen.

3 minute read

In short
  • Fallbacks keep the product working when a model or provider fails.
  • Two independent 99% providers together reach about 99.99%.
  • Retry with backoff and jitter, then switch model, then degrade gracefully.

In plain words

A fallback is what happens when the first choice fails: a retry, another model, another provider, or a simpler non-AI answer. Without one, a provider's bad hour becomes your product's outage.

Worked example: why a second provider helps

  1. 1
    One provider, 99% available

    1% of a 720-hour month is down time.

    about 7.2 hours a month unavailable
  2. 2
    Two independent providers

    Both must fail at once to cause an outage.

    1 − 0.01 × 0.01 = 99.99% available
  3. 3
    What that means

    If their outages really are independent.

    about 4.3 minutes a month unavailable

A sensible order of fallbacks

stepwhennote
Retry with backoffBrief errors and rate limitsWait 1 s, then 2 s, then 4 s, with a little randomness
Another model or providerRepeated failures or slownessTest it on your evals first; prompts may need tweaks
Degrade gracefullyEverything AI is downShow cached answers, a search result or 'try again soon'
Your turn

Why add randomness to retry delays instead of everyone retrying after exactly 1 second?

What to remember

Set timeouts so a stuck call doesn't hang a user. Keep fallback models tested; a fallback that silently gives worse answers can be worse than an honest error. Alert when fallbacks fire often, because that's a problem to fix, not a normal state.

Common mistakes

  • Retrying instantly in a tight loop during an outage, which makes the overload worse.
  • Falling back to a different model without testing that your prompts work on it.
  • Showing users a raw error instead of a useful degraded experience.
Check yourself

Answer in your own words, out loud or on paper, before you open the model answer.

How would you make an LLM feature resilient to provider outages?

I'd plan for the provider being slow or down. Every call gets a timeout. Transient errors get a small number of retries with exponential backoff and jitter. If a provider keeps failing, a circuit breaker stops sending it traffic for a while so we fail fast instead of piling up requests. Then a fallback chain: a second model or provider that's been tested with the same prompts and evaluation set, then a simpler path like cached answers or a non-AI feature, and finally a clear message to the user with a way to continue, like contacting a person. I'd monitor error rates and fallback usage, and run occasional drills to make sure the fallbacks actually work.

More to think about
  • What is exponential backoff with jitter?
  • What are the risks of falling back to a different model?

Go further

Was this lesson clear?
Next: Budgets →