Learn · Module 13 · Production AI and MLOps · Lesson 2 of 7
Fallbacks
Every AI provider has outages, rate limits and slow days. Plan for them before they happen.
- Fallbacks keep the product working when a model or provider fails.
- Two independent 99% providers together reach about 99.99%.
- Retry with backoff and jitter, then switch model, then degrade gracefully.
In plain words
A fallback is what happens when the first choice fails: a retry, another model, another provider, or a simpler non-AI answer. Without one, a provider's bad hour becomes your product's outage.
Worked example: why a second provider helps
- 1One provider, 99% available
1% of a 720-hour month is down time.
about 7.2 hours a month unavailable
- 2Two independent providers
Both must fail at once to cause an outage.
1 − 0.01 × 0.01 = 99.99% available
- 3What that means
If their outages really are independent.
about 4.3 minutes a month unavailable
A sensible order of fallbacks
| step | when | note |
|---|---|---|
| Retry with backoff | Brief errors and rate limits | Wait 1 s, then 2 s, then 4 s, with a little randomness |
| Another model or provider | Repeated failures or slowness | Test it on your evals first; prompts may need tweaks |
| Degrade gracefully | Everything AI is down | Show cached answers, a search result or 'try again soon' |
Why add randomness to retry delays instead of everyone retrying after exactly 1 second?
What to remember
Set timeouts so a stuck call doesn't hang a user. Keep fallback models tested; a fallback that silently gives worse answers can be worse than an honest error. Alert when fallbacks fire often, because that's a problem to fix, not a normal state.
Common mistakes
- Retrying instantly in a tight loop during an outage, which makes the overload worse.
- Falling back to a different model without testing that your prompts work on it.
- Showing users a raw error instead of a useful degraded experience.
Answer in your own words, out loud or on paper, before you open the model answer.
How would you make an LLM feature resilient to provider outages?
I'd plan for the provider being slow or down. Every call gets a timeout. Transient errors get a small number of retries with exponential backoff and jitter. If a provider keeps failing, a circuit breaker stops sending it traffic for a while so we fail fast instead of piling up requests. Then a fallback chain: a second model or provider that's been tested with the same prompts and evaluation set, then a simpler path like cached answers or a non-AI feature, and finally a clear message to the user with a way to continue, like contacting a person. I'd monitor error rates and fallback usage, and run occasional drills to make sure the fallbacks actually work.
- What is exponential backoff with jitter?
- What are the risks of falling back to a different model?
Go further
- ArticleTimeouts, retries and backoff with jitterAmazon Builders' Library
- ArticleCircuit BreakerMartin Fowler