1Context and scope
A chat product calls a hosted model through its own AI gateway (the LiteLLM or Portkey kind): one place for keys, routing, retries and logging. An exact-match prompt cache answers repeated prompts. Hosted models fail in ways more servers cannot fix: quota errors (429), overload, timeouts and outages. Halfway through the run the provider has a 15-second outage.
2Goals and non-goals
Goals
- 40 chats a second; 1,500 prompt tokens and 350 answer tokens each.
- Success at least 99% through a 15-second provider outage.
- Time to first token p99 at most 2.5 s (people see the answer start streaming).
Non-goals
- Making the backup model give identical answers: prompts may need tuning per model.
- Training or hosting the model yourself (see the self-hosted alternative below).
- Cross-region data residency for prompts.
3Capacity estimate
Back-of-the-envelope numbers from the template. The simulation uses the same ones, so you can check them against a run.
| Quantity | Value | How |
|---|---|---|
| Calls reaching the model | 32/s | 40/s × (1 − 20% prompt-cache hits) |
| Primary quota | 3,000/min (50/s) | headroom of about 1.5× |
| Model response time | ≈ 4.8 s | 450 ms to first token + 349 tokens ÷ 80 tokens/s |
| Backup response time | ≈ 6 s | 650 ms to first token, 65 tokens/s |
| Chats lost to the outage | ≈ 480 | 15 s × 32/s, with no fallback |
| Cost per chat | ≈ 1.0¢ (backup 1.1¢) | 1,500 × $3/M + 350 × $15/M |
4The design
Each component in the template, in the order a request meets it, and why it is there.
- Load balancer
Where chat traffic enters.
- AI gateway
Your service in front of every model: holds the provider keys, applies per-team limits, logs tokens and cost, and decides which model takes each call.
- Prompt cache
Answers prompts it has seen before, word for word, without a model call. Cheap and fast, but only helps for repeated prompts.
- Primary model (provider A)
The model you prefer: best quality for the price. Its quota and availability are outside your control.
- Fallback model (provider B)
Added by the fix: a second provider (or region) that takes only the calls the primary fails. It is a little slower and pricier, and only paid for when used.
5Key concepts
Fallback, not retry
Retrying the same provider during an outage just fails again (and adds load when it is overloaded). A fallback sends the failed call somewhere else. Gateways usually retry once for brief errors, then fall back, with a per-attempt timeout so a hung call does not eat the whole budget.
Which errors to fall back on
429 (quota), 5xx and overload errors, and timeouts are worth falling back on. 400-type errors (a bad request, a prompt too long) fail on every provider, so falling back only doubles the cost. In the lab a fallback can be limited to some failures.
Time to first token vs response time
Model answers stream. Time to first token is how long until text starts appearing; response time adds the decoding of every output token. Users judge the first; your capacity and timeouts must allow for the second.
Circuit breaking
When a provider is clearly down, real gateways stop sending to it for a cooldown and go straight to the backup, so users skip the failed first attempt. The lab models the failed attempt, so you see the cost of not having a breaker.
6Alternatives considered
- A second provider adds resilience but answers differ between models; keep prompts and output checks per model.
- Fallback only on failure keeps cost low; splitting steady traffic across providers keeps both warm and quotas in use, but you pay the pricier one all the time.
- Self-hosted GPUs as the fallback avoid a second vendor but cost money while idle.
- A longer per-attempt timeout gives a slow provider a chance; a shorter one reaches the backup sooner.
7Failure modes and fixes
What goes wrong in the simulation, why, and the change that fixes it. Each one shows up as a finding in the lab; ask “what broke?” to see it, and for the fix when you want it.
A quarter of chats fail during the outage
WhyEvery call to provider A fails and nothing else can answer. More gateway replicas or streams cannot help: the problem is outside your system.
FixAdd a fallback to a second provider. The run then shows "Fell back" on the primary and "Took over" on the backup.
Quota errors (429) at 3x traffic
Why96 calls a second against a 50-a-second quota. The fallback absorbs the overflow until its own quota runs out.
FixRaise the prompt-cache hit rate, ask the provider for more quota, or spread steady traffic over both providers instead of only on failure.
Slower answers during the outage
WhyEach fallen-back chat waits for the failed first attempt, then the backup, which is slower.
FixA circuit breaker on the primary, and a backup sized for the full load if the primary can be down for long.
8Try it in the lab
Open the template, then type these into the copilot one at a time. The first few are also offered as buttons when the template opens.
what broke?The provider outage at 20 s, and that there is no fallback.
fix itAdds a backup provider with a dashed fallback line; the rerun keeps success near 100%.
prompt cache hit rate 50%Fewer calls reach the model: lower cost and more quota headroom.
traffic spike 3xQuota errors on the primary; the fallback takes the overflow until it runs out too.
how much does it costMonthly cost, with the model tokens the biggest line.
9Check yourself
- Why does doubling the AI gateway replicas not help during the outage?
- Which errors should trigger a fallback, and which should not?
- At 32 calls a second, how big must the backup quota be to survive a full outage of the primary?
- What does a circuit breaker change for the user during a long outage?