Problems / LLM serving, medium

Decode is the bottleneck

Disaggregated serving with a small decode pool: answers got longer and decode slots fill up.

Free to open. Runs in your browser, no sign-up.

Your goals

Success ≥ 99%, p99 ≤ 5 s.

120 requests a second for 30 seconds: 100% request (reads).

What each component does in this design, and what calls it. The simulator mimics each one from these settings.

Component	Its job here	Settings
Load balancerLoad balancer	Traffic enters here.Calls AI app.Layer 4 balancers route by IP and port; layer 7 balancers read HTTP and can route by path or header.	–
AI appApp service	Load balancer forwards to it.Calls Prefill GPU pool, Decode GPU pool.Little's law: throughput ≈ concurrency ÷ latency. 200 slots at 50 ms serve about 4,000 req/s.	3 replicas × 500 slots · timeout 30 s
Prefill GPU poolPrefill GPU pool	AI app calls it.Splitting prefill (reading the prompt) from decode (writing the answer) onto separate GPUs stops long prompts from stalling everyone else’s output.	128 slots
Decode GPU poolDecode GPU pool	AI app calls it.Decoding is memory-bound: each request holds its KV cache in GPU memory for as long as it generates.	128 slots

Try it first. Open one hint at a time if you get stuck.

Hint 1

Decode slots are held for the whole answer.

Hint 2

Slots needed = rate × decode time.

One way to solve it

Add capacity to the decode pool.