Design doc · 11 of 12

MCP agent platform

Agents that think with a model and act through MCP tools: steps multiply calls, errors and cost.

Template
MCP agent platform
Level
Advanced
Reading time
15 min
Topics
MCPAgent loopTool serversCompounding errorsAgent memoryCost per task

The challenge in the lab 4 tasks/s. The agent thinks (model) and acts (tools through an MCP gateway) 6 times per task, with vector memory and results in S3. Keep success ≥ 95% and p99 ≤ 60 s. A 1% tool error compounds over the steps: find it and cut it, and watch the cost.

1Context and scope

An agent API takes tasks ("triage this ticket", "update the report"). An orchestrator runs the agent loop: ask the model what to do, call a tool through an MCP gateway, repeat. Tools are MCP servers (search, CRM, code). The agent reads and writes vector memory, and results are stored in S3. Each task takes six steps.

2Goals and non-goals

Goals

  • 4 tasks a second, six think-and-act steps each.
  • Success at least 95% of tasks; p99 at most 60 s.
  • Know the cost of a task, not just of a call.

Non-goals

  • Making the agent smarter (prompting and evaluation are separate work).
  • The approval UI for risky actions (a human-approval step would sit before those tools).
  • Writing the MCP servers themselves.

3Capacity estimate

Back-of-the-envelope numbers from the template. The simulation uses the same ones, so you can check them against a run.

QuantityValueHow
Model calls24/s4 tasks/s × 6 steps
Tool calls24/sone per step, through the MCP gateway
Task success with a 1% tool error≈ 94%0.99⁶; the agent gives up on a failed step
Time per task≈ 20 s6 × (≈ 3.3 s model response + tool time)
Model quota1,200/min (20/s)six steps at full rate would exceed it
Tokens per task≈ 18,000 in6 × 3,000 prompt tokens (the context grows each step)

4The design

Each component in the template, in the order a request meets it, and why it is there.

  1. Load balancer and Agent API

    Accept tasks, authenticate, and return or store the result.

  2. Agent orchestrator

    Runs the loop: model call, tool call, repeat until done or out of steps. Steps run one after another because each acts on the last result.

  3. Model

    Decides the next action from the task, the tool results and memory. Its response time is most of each step.

  4. MCP gateway

    One place for tool auth, the tool registry, per-tool limits and audit logs, so the agent never holds raw credentials.

  5. MCP tool servers

    Expose tools (search, CRM, code) over the Model Context Protocol. Each call can fail or be slow.

  6. Agent memory (vector)

    Past results and facts the agent retrieves each step.

  7. S3 (task results)

    Stores the final output and artifacts: cheap, durable, and outside the hot path.

5Key concepts

MCP: host, client, server

In the Model Context Protocol the host (your agent app) creates a client per server; each server exposes tools, resources and prompts. Servers cannot see the whole conversation or each other, which keeps a misbehaving tool contained. A gateway in front centralises auth and policy.

Steps multiply everything

Each step is a model call and a tool call. Six steps mean six times the model load, quota use and tokens, and six chances to fail. Per-call reliability of 99% becomes 94% per task.

Steps are sequential

An agent cannot run its steps in parallel: step 3 needs the result of step 2. So the levers are fewer steps (better tools, better plans, step limits), faster models, and retrying failed tool calls.

Cost per task

Prompt tokens grow as tool results pile into the context. Prompt caching of the stable prefix (system prompt, tool definitions) and loading tools on demand keep it in check.

6Alternatives considered

  • More steps can solve harder tasks but cost more, take longer and fail more often.
  • A gateway in front of MCP servers adds a hop but centralises auth, limits and audit.
  • Retrying tool calls rescues tasks but can repeat side effects; only retry idempotent tools or use idempotency keys.
  • Loading every tool definition into each prompt is simple; loading tools on demand saves tokens on large tool sets.

7Failure modes and fixes

What goes wrong in the simulation, why, and the change that fixes it. Each one shows up as a finding in the lab; ask “what broke?” to see it, and for the fix when you want it.

About 6% of tasks fail

WhyA 1% tool error rate compounds over six steps (1 − 0.99⁶).

FixCap the agent at fewer steps; retry idempotent tool calls once in the MCP gateway; make tools return clearer errors so the agent can recover.

Quota errors as traffic grows

WhyEvery task is six model calls; the quota counts calls, not tasks.

FixFewer steps, prompt caching, a bigger quota, or a fallback model for overflow.

Slow tasks

WhySteps add up: six model responses of about 3.3 s plus tool time.

FixFewer steps, a faster model for simple steps (a model router), and fast tools.

8Try it in the lab

Open the template, then type these into the copilot one at a time. The first few are also offered as buttons when the template opens.

  1. what broke?

    Tasks failing from tool errors compounding over the steps.

  2. cap the agent at 3 steps

    Model and tool calls halve, success rises, and the cost per month drops.

  3. how much does it cost

    Model tokens dominate; compare before and after capping steps.

  4. stress test

    Shows the task rate where the model quota or tool servers give out.

Open “MCP agent platform” in the lab

9Check yourself

  1. Why is a 1% tool error rate a 6% task error rate here?
  2. Why can’t the orchestrator run its six steps in parallel?
  3. What does an MCP gateway give you that calling MCP servers directly does not?
  4. Which tool calls are safe to retry automatically?

10Further reading