1Context and scope
A support assistant answers questions from your documentation. Each question is embedded, similar passages are found in a vector database, and a large language model writes the answer from them. A semantic cache returns stored answers to questions it has seen before.
2Goals and non-goals
Goals
- 40 questions a second, 4,000-token prompts.
- Success at least 99% and time to first token p99 at most 1.5 s.
- A provider quota of 6 million input tokens a minute.
Non-goals
- Training or fine-tuning a model.
- Answer quality: retrieval and answers are evaluated separately; this design is about load, latency and cost.
- Conversation memory across sessions.
3Capacity estimate
Back-of-the-envelope numbers from the template. The simulation uses the same ones, so you can check them against a run.
| Quantity | Value | How |
|---|---|---|
| Questions reaching the model | 30/s | 40 × (1 − 25% cache hits) |
| Input tokens | 7.2M a minute | 30 × 4,000 × 60: 20% over the 6M quota |
| Cache hit rate that fits | ≥ 38% | 40 × 0.62 × 4,000 × 60 ≈ 6M |
| Vector index | ≈ 9 GB | 1M vectors × 1,536 dims × 4 bytes × 1.5 overhead |
4The design
Each component in the template, in the order a request meets it, and why it is there.
- AI gateway
One place for keys, quotas, retries and logging in front of the model providers.
- Semantic cache
Matches questions by meaning; a hit skips retrieval and the model entirely.
- Embedding model
Turns the question into a vector for search.
- Vector DB
Finds the most similar passages; its index must fit in memory.
- LLM endpoint
Generates the answer from the passages. Quotas, time to first token and cost live here.
5Key concepts
Quotas are a hard ceiling
A provider limits tokens per minute. Above it, calls fail with 429 no matter how many servers you run. Capacity planning for AI is token planning.
Prompt size is a lever
Tokens per question × questions per minute is the load. Retrieving fewer, better passages (a reranker) shrinks prompts and cost as well as quota use.
Semantic caching
Embedding the question and matching similar past questions turns repeats into near-free answers. Tune the similarity threshold: too loose returns wrong answers.
Time to first token
Users judge streaming answers by when text starts to appear. Queueing for a stream slot or a quota adds directly to it.
6Alternatives considered
- A looser semantic cache saves tokens but risks wrong answers.
- Shorter prompts are cheaper but may drop context the model needed.
- A bigger quota costs money and may need provider approval.
7Failure modes and fixes
What goes wrong in the simulation, why, and the change that fixes it. Each one shows up as a finding in the lab; ask “what broke?” to see it, and for the fix when you want it.
Calls to the model fail with quota errors (429) and goals are missed.
WhyAbout 7.2M input tokens a minute against a 6M quota.
FixRaise the cache hit rate, shorten prompts, or raise the quota. More replicas of your own services do nothing.
8Try it in the lab
Open the template, then type these into the copilot one at a time. The first few are also offered as buttons when the template opens.
cache hit rate 40%Tokens fall under the quota; 429s disappear.
scale the API to 6 replicasNo change: the bottleneck is the provider quota.
what broke?The quota finding and its fix.
stress testHow many questions a second the quota allows.
9Check yourself
- Compute the tokens a minute for 50 questions a second at 3,000 tokens with a 30% cache hit rate.
- Why does adding replicas not fix a quota problem?
- What are two ways to reduce prompt size without hurting answers?