1Context and scope
An agent handles tasks by alternating model calls and tool calls until it is done. Each task takes 8 steps. The model allows 4,000 calls a minute and 120 answers streaming at once, so a modest task rate becomes a large call rate, and cost grows with every step.
2Goals and non-goals
Goals
- 15 tasks a second, 8 model and tool calls each.
- Success at least 97% (critical), p99 at most 20 s, cost at most $1M a month.
Non-goals
- Making the agent smarter: the lesson is bounding what it can call and spend.
- Whether each tool returns the right answer.
- The user interface for approving actions.
3Capacity estimate
Back-of-the-envelope numbers from the template. The simulation uses the same ones, so you can check them against a run.
| Quantity | Value | How |
|---|---|---|
| Model calls | 7,200 a minute | 15 × 8 × 60: 1.8× the 4,000 quota |
| Steps that fit the quota | ≤ 4 | 15 × 4 × 60 = 3,600 |
| Streams in use | ≈ 430 at 8 steps | 120 calls a second × about 3.6 s per answer; the model allows 120 |
| Steps that fit the streams | ≤ 2 | 15 × 2 × 3.6 s ≈ 108 |
| Cost at 2 steps | ≈ $1M a month | 3,000 input and 250 output tokens per call at $3 and $15 per million tokens |
| Latency per task | seconds | 8 steps × (time to first token + generation + tool) |
4The design
Each component in the template, in the order a request meets it, and why it is there.
- Gateway
Entry point; where per-tenant budgets belong.
- Agent orchestrator
Runs the loop: decide, call a tool, read the result, repeat until done or out of steps.
- LLM endpoint
Every step is a model call: quota, latency and cost scale with steps.
- Tool server (MCP)
The actions the agent can take. Each call is another round trip.
5Key concepts
Agents multiply load
One task is many model calls. Capacity planning is tasks × steps, not tasks. A small change in average steps is a large change in quota use and cost.
Step limits
Cap steps per task (often 10 to 20; here 4 fit the quota but only 2 fit the streams) and return the best answer so far when the cap is hit. Runaway loops are the costliest failure in agent systems.
Prompt caching
Agent prompts repeat a long prefix every step (instructions, tool definitions, history). Caching that prefix cuts cost and time to first token.
Budgets in two places
A per-task budget in the agent and a per-tenant budget at the gateway, so one runaway task or customer cannot drain the quota for everyone.
6Alternatives considered
- Fewer steps mean faster, cheaper tasks but can lower answer quality on hard tasks.
- A bigger quota fixes errors but not cost or latency.
7Failure modes and fixes
What goes wrong in the simulation, why, and the change that fixes it. Each one shows up as a finding in the lab; ask “what broke?” to see it, and for the fix when you want it.
Quota errors, slow tasks and a cost over budget.
Why7,200 model calls a minute against a 4,000 quota, and about 430 answers streaming against 120; every step adds latency and cost.
FixCap the agent at 2 steps (4 fits the quota but not the streams), add a prompt cache, or buy more quota and streams, which raises the cost.
8Try it in the lab
Open the template, then type these into the copilot one at a time. The first few are also offered as buttons when the template opens.
cap the agent at 4 stepsCalls fit the quota, but the model still runs out of streams: success stays low.
cap the agent at 2 stepsFits the quota and the streams: success above 97% for about $1M a month.
add a prompt cacheLower cost and faster first tokens per step.
what broke?The quota and fan-out findings with their fixes.
how much does it costCompare the monthly cost with the $1M budget.
9Check yourself
- How many model calls a minute do 20 tasks a second at 6 steps make?
- Why is a step limit both a cost control and a reliability control?
- What part of an agent prompt is a good candidate for prompt caching?