Learn / Questions
Tokens per minute is a ceiling on the provider’s side. Replicas on your side cannot raise it; they just queue behind it faster.
Most model providers meter in tokens per minute. A support assistant that sends 4,000-token prompts at 40 questions a second needs 4,000 × 40 × 60, about 9.6 million input tokens a minute, before counting the answers. Against a 6 million tokens-per-minute quota, more than a third of requests are rejected with a rate-limit error, no matter how many API servers sit in front of the model.
Teams reach for the usual fix, more replicas, and it changes nothing, because the bottleneck is not CPU or slots. It is a budget set by someone else. The symptoms are rate-limit errors, climbing time to first token as requests wait for the window to roll over, and retries that spend the same quota twice.
The only levers are on the token side: send fewer tokens per request, answer some questions without the model, spread the load across more than one provider or model, or buy a bigger quota.
| Questions per second | 40 |
|---|---|
| Prompt size | 4,000 tokens |
| Tokens needed per minute | ≈ 9.6 million input |
| Quota | 6 million tokens/min |
| Requests over quota | more than a third |
Raise the answer-cache hit rate so repeated questions never reach the model, shorten prompts (retrieve fewer, better chunks; cache the fixed prefix), add a second model or provider as a fallback, and keep retries from spending quota twice.
Opening and running a problem is free. Changes by command cost 1 credit each; you have 25 free credits a day.