Learn / Questions
A slow dependency plus a timeout and two retries means three times the load, exactly when it can least take it.
Retries are written for the case where a request fails on its own: a dropped packet, one bad instance. Retry and it works. During an overload, every request is failing for the same reason, and the retry does not change that reason. It adds another request to the queue that caused the failure.
Take a database that has slowed to 9 seconds per read, behind an API that gives up after 5 seconds and retries twice. Every user request now becomes three database queries, none of which finish before the API stops waiting. The database works through a queue of abandoned work while new work arrives at three times the rate. Its latency rises further, which makes more callers time out, which produces more retries. This is the retry storm: a positive feedback loop between a slow dependency and the clients trying to help.
The tell is a database that stays overloaded after the original cause is gone. The clients keep it there.
| Database read time | 9 s |
|---|---|
| API timeout | 5 s |
| Retries per request | 2 |
| Database load per user request | 3 queries, all abandoned |
Fix the dependency first (the 9 ms preset shows what a healthy database does), then make retries safe: a budget (retry at most a small share of requests), exponential backoff with jitter, and a circuit breaker that stops calling a dependency that is already failing.
Opening and running a problem is free. Changes by command cost 1 credit each; you have 25 free credits a day.