Learn / Questions
A server has a fixed number of request slots and CPUs. Past that line, requests queue, and queues only grow.
A single API replica can hold a fixed number of requests at once (its slots, or threads) and has a fixed amount of CPU. Each request occupies a slot for as long as it takes and burns some CPU time. Multiply arrival rate by time per request and you get how many slots are busy on average. That is Little’s law, and it decides everything.
Suppose the server has 150 slots and two CPUs, and launch day brings 1,500 requests a second. If each request takes 100 ms, 150 slots are busy on average, which is every slot, with no room for the variation that real traffic has. Requests arriving in a burst find no free slot and wait in a queue. Waiting makes them take longer, which keeps slots busy longer, which makes the next arrivals wait more. Once the arrival rate is above what the server can finish per second, the queue does not settle; it grows until requests time out.
Nothing is broken. The server is simply asked for more than it can finish, and a queue is where the difference goes.
| Traffic | 1,500 requests/s |
|---|---|
| Slots per replica | 150 |
| Busy slots at 100 ms per request | 1,500 × 0.1 s = 150, every one of them |
| Replicas that leave headroom | 3 to 4 |
Stateless services scale out: add replicas behind the load balancer until the busiest one sits well under its slots and CPU at peak, and keep an autoscaler or a rate limiter for the day traffic beats the estimate.
Opening and running a problem is free. Changes by command cost 1 credit each; you have 25 free credits a day.