1Context and scope
An API reads from Postgres. The query is missing an index, so each read takes about 9 seconds. The API gives up after 5 seconds and retries twice. Nothing succeeds, and the retries make it worse.
2Goals and non-goals
Goals
- 2,000 reads a second.
- Success at least 99% and Apdex at least 0.90.
Non-goals
- Fixing the query itself: the lesson is how callers protect themselves while a dependency is slow.
- Adding a cache to hide the slow reads.
- Changing the database engine.
3Capacity estimate
Back-of-the-envelope numbers from the template. The simulation uses the same ones, so you can check them against a run.
| Quantity | Value | How |
|---|---|---|
| Read latency | 9 s | p50; p99 about 30 s |
| API timeout | 5 s | shorter than the median, so almost every call times out |
| Attempts per request | 3 | 1 + 2 retries: up to 6,000 queries/s offered |
| Database capacity | ≈ 5.5 queries/s | Little’s law: 50 connections ÷ 9 s |
| Capacity at 9 ms | ≈ 5,500 queries/s | 50 ÷ 0.009 s: a thousand times more |
4The design
Each component in the template, in the order a request meets it, and why it is there.
- Load balancer
Entry point; spreads requests over the API replicas.
- API (3 replicas, 5 s timeout, 2 retries)
Calls Postgres for every request. Its timeout and retry policy decide how it behaves when Postgres is slow.
- Postgres (50 connections)
Each connection serves one query at a time; slow queries hold connections and everything else queues.
5Key concepts
Little’s law
Concurrency = throughput × latency. With 50 connections and 9 s per query, the database can complete only about 5.5 queries a second. Latency is capacity: making the query 1,000× faster makes the database 1,000× bigger.
A timeout shorter than the median never succeeds
If the dependency usually takes 9 s and you wait 5 s, almost every call fails. Timeouts should sit above the dependency’s normal tail latency and below what the user will wait.
Retry amplification
Each retry is another query on a database that is already overloaded. Two retries can triple the load exactly when there is no capacity to spare. Retries help with rare, random failures, not with overload.
Retry budgets and backoff with jitter
Cap retries to a small share of traffic (for example 10%), back off exponentially and add jitter so retries do not arrive in synchronized waves.
6Alternatives considered
- A longer timeout lets slow calls finish but ties up API slots longer; a shorter one fails fast but may give up on calls that would have succeeded.
- Retries hide brief failures from users but amplify overload; budgets keep the benefit without the danger.
7Failure modes and fixes
What goes wrong in the simulation, why, and the change that fixes it. Each one shows up as a finding in the lab; ask “what broke?” to see it, and for the fix when you want it.
Success near 0%; Postgres connections and queue full; retries multiply the load.
WhyEvery call outlives the 5 s timeout, the retries add more queries, and the queue never drains.
FixFix the cause: index the query (9 ms). Then set a sane timeout and a retry budget so a future slowdown cannot triple the load.
8Try it in the lab
Open the template, then type these into the copilot one at a time. The first few are also offered as buttons when the template opens.
make the database respond in 9 msSuccess jumps to ~100%: latency was the capacity problem.
set the API timeout to 2 sFails faster but still fails while reads take 9 s.
no retriesLess load on Postgres, same failures: retries were making it worse.
what broke?The findings name the timeout below latency and retry amplification.
9Check yourself
- Using Little’s law, how many queries a second can 50 connections serve at 200 ms each?
- Why do retries make an overload worse, and when do they help?
- Where should a timeout sit relative to the dependency’s p50 and p99?