System design for AI infrastructure

System design interviews now ask how you would serve a model, build retrieval, or keep an agent from running away with the bill. Here you do it hands-on: 100 AI infrastructure problems, plus 11 AI systems problems and guided templates, where an LLM service, a RAG pipeline or an agent platform breaks under real traffic and you fix it.

What you will run into

LLM serving
Token and request quotas, concurrent streams, time to first token, prefill and decode GPU pools, KV cache reuse, speculative decoding, multi-LoRA, provider outages and fallback.
RAG and retrieval
Embedding quotas shared with ingestion, vector indexes that outgrow memory, rerankers, query rewriting, GraphRAG, semantic caching and prompt size.
Agents and tools
Steps that multiply load and cost, MCP tool servers and gateways, sandboxes, human approval queues, checkpoints, memory and tool results worth caching.
Multi-agent and harnesses
Subagent fan-out, lead agents that over-plan, long-running coding harnesses, compaction and shared tools.
Security and governance
Prompt injection in support bots, email and web pages, guardrails that block real users, spend governors and audit logs.

54 AI and agent components back these problems, including semantic cache, embedding model, vector DB, LLM endpoint, AI gateway, prompt cache, reranker, guardrails and more.

Start with a guided system

All AI problems

111 problems

LLM serving 20

  1. Token quota at launchA chat product sends 2,000-token prompts at 50 requests a second. The provider allows 4 million input tokens a minute.EasySolve
  2. Not enough concurrent streamsEach answer streams for about 4 seconds, and the deployment allows 100 concurrent streams. Traffic is 40 requests a second.EasySolve
  3. Provider outage at peakAt 15 s the only model provider goes down for 15 seconds, at 30 chats a second.MediumSolve
  4. Timeouts on long answersAnswers are 800 tokens at 70 tokens a second, about 12 seconds each. The app gives up after 8 seconds.EasySolve
  5. Prompt cache that rarely hitsAn exact-match prompt cache sits in front of the model but hits only 5%: timestamps in the prompt make every prompt unique. The model’s token quota is tight.MediumSolve
  6. Self-hosted GPUs at capacityYou serve an open-weights model on your own GPU pool with 32 slots. Requests take about a second; traffic is 50 a second.EasySolve
  7. GPU autoscaling ceilingThe GPU pool autoscales from 2 to 3 nodes of 16 slots each, but traffic needs more than 3 nodes.MediumSolve
  8. Prefill is the bottleneckServing is split into prefill and decode pools (disaggregated serving). Prompts got longer: the prefill pool can’t keep up, and time to first token suffers.MediumSolve
  9. Decode is the bottleneckDisaggregated serving with a small decode pool: answers got longer and decode slots fill up.MediumSolve
  10. KV cache not reusedA KV-cache tier sits in front of prefill, but only 10% of requests reuse a cached prefix: the system prompt changes per user, so prefixes never match.HardSolve
  11. Draft model can’t keep upSpeculative decoding speeds up answers, but the single draft-model GPU is now the bottleneck at 150 requests a second.MediumSolve
  12. Too many fine-tunes on one GPUOne multi-LoRA server hosts 200 customer adapters on a shared base model. At 60 requests a second the shared batches overflow.MediumSolve
  13. AI gateway key limitAll traffic goes through an AI gateway whose API key allows 1,500 requests a minute. Traffic is 40 a second.EasySolve
  14. PII redaction on the hot pathEvery prompt is scanned by a PII redactor with 2 vCPUs and 10 ms of CPU per prompt, at 400 requests a second.MediumSolve
  15. An LLM judge on every answerEvery answer is graded inline by an LLM judge with 20 slots before it returns. The judge is now the slowest part.MediumSolve
  16. Tracing drops spansEvery request writes a trace to an observability service limited to 2,000 spans a minute. At 60 requests a second, traces are rejected and requests fail with them.EasySolve
  17. Voice assistant: transcription backs upA voice assistant transcribes speech, answers, and speaks the reply. Speech-to-text has 16 slots at 30 utterances a second.MediumSolve
  18. Image generation queueAn image generator takes 4 seconds per image on 40 slots. Users request 15 images a second.EasySolve
  19. Retries into a rate limitThe app retries every failed model call 3 times immediately. When the token quota is hit, retries multiply the 429s.HardSolve
  20. Batch inference used for live trafficLive requests were routed to a batch-inference queue meant for offline jobs: it holds requests for 30+ seconds.HardSolve

RAG and retrieval 16

  1. Query rewriting on every questionAgentic RAG rewrites each question with a small model before retrieval. The rewriter has 8 slots at 80 questions a second.MediumSolve
  2. GraphRAG traversals melt the CPUMulti-hop questions traverse a knowledge graph: 10 ms of CPU each on 8 vCPUs, at 1,200 questions a second.MediumSolve
  3. Hybrid search saturatedRetrieval combines keyword and vector search. The hybrid search service has 40 slots at 1,500 queries a second.EasySolve
  4. Reranker on a single GPUA cross-encoder reranks the top 50 chunks for every question on one GPU with 16 slots, at 300 questions a second.EasySolve
  5. Embedding calls queue upEvery question is embedded first, but the embedding deployment allows 8 concurrent calls at 400 questions a second.EasySolve
  6. Semantic cache too strictA semantic cache in front of a RAG assistant hits 10%: its similarity threshold is 0.99. The model quota can’t take the rest.MediumSolve
  7. Context assembly is slowA context builder stitches retrieved chunks, history and instructions for each question. It has 40 slots at 1,000 questions a second.MediumSolve
  8. OCR before every answerA document assistant OCRs scanned pages on demand: 12 slots, 1.5 seconds per page, 15 pages a second.MediumSolve
  9. RAG answers stop during an outageRetrieval works, but the answer model’s provider is down from 15 s to 30 s.MediumSolve
  10. Too many chunks in the promptThe assistant stuffs 6,000 tokens of chunks into every prompt at 30 questions a second, against an 8-million-token-a-minute quota.MediumSolve
  11. Corrective RAG evaluator bottleneckCorrective RAG grades retrieved chunks with an evaluator model before answering. The evaluator has 20 slots and grades every question.HardSolve
  12. Retrieval traces over quotaEach RAG question writes retrieval traces to an observability API capped at 3,000 a minute, at 80 questions a second.EasySolve
  13. Multi-hop retrieval loopsAn agentic retriever makes up to 8 search hops per question, and each hop hits the search service.HardSolve
  14. Rerank results never reusedA cache in front of the reranker hits 15%, because it is keyed on the raw question text including whitespace and casing.MediumSolve
  15. GraphRAG index shared with ingestionThe GraphRAG index answers questions and absorbs entity extraction from new documents at the same time, at 1,500 operations a second.HardSolve
  16. Scrapers hammer the RAG APIAt 8 s, 3,000 req/s of scrapers start querying your RAG assistant, eating model capacity meant for users.MediumSolve

Agents and tools 22

  1. An agent that takes 12 stepsA support agent takes 12 model-and-tool steps per ticket. Tools fail 1% of the time, and every failure sinks the whole task.EasySolve
  2. MCP gateway quotaEvery tool call goes through an MCP gateway (auth, allow-lists, quotas) limited to 600 calls a minute. Agents make about 18 tool calls a second.EasySolve
  3. One slow tool serverA CRM tool server with 8 slots takes 300 ms per call. Agents call it every step.EasySolve
  4. Code sandboxes run outA data-analysis agent runs Python in sandboxes: 6 sandboxes, about 1.2 s per run, 4 runs per task at 2 tasks a second.MediumSolve
  5. Browser automation bottleneckA web agent drives headless browsers: 16 browsers, about 2.5 s per action, 3 actions per task at 3 tasks a second.MediumSolve
  6. Computer-use desktops are scarceA computer-use agent clicks through a legacy app on virtual desktops: 10 desktops, about 3 s per action, 4 actions per task.MediumSolve
  7. Approvals pile upA finance agent asks a human to approve every payment. Five reviewers take about 20 s each; tasks arrive at 0.5 a second.MediumSolve
  8. Agent memory on a small storeThe agent reads and writes long-term memory every step, on a memory store with 2 slots and 60 ms per call.EasySolve
  9. Same tool call, every stepAgents repeat the same read-only lookups, but the tool-result cache hits 10% because it is keyed on raw arguments with timestamps.MediumSolve
  10. Checkpointing every stepDurable execution saves the agent’s state after every step to a checkpoint store with 10 slots and 50 ms writes.MediumSolve
  11. Policy check on every tool callEvery tool call is authorised by a policy engine with 8 slots and 30 ms per check.EasySolve
  12. Tool search on every stepWith 400 tools, the agent searches for the right few each step. The tool-search service has 6 slots.MediumSolve
  13. Compaction calls overflowLong sessions are compacted every step by a summariser with 20 slots, about 600 ms each.MediumSolve
  14. Agents burn the token quotaEach step sends 3,000 tokens of context. Agents make 24 model calls a second against a 3-million-token-a-minute quota.MediumSolve
  15. Model outage mid-taskThe agent’s model provider goes down for 15 seconds. Every task in flight fails at its next step.MediumSolve
  16. Agent model out of streamsAgents hold a model stream for about 2 s per step. The deployment allows 30 streams; agents need about 50.EasySolve
  17. Partner agent goes offlineYour travel agent books hotels through a partner’s agent over A2A. The partner is down from 15 s to 35 s.MediumSolve
  18. Partner agent at capacityA partner agent you delegate to over A2A handles 20 tasks at once, about 2.5 s each. You send it 12 a second.MediumSolve
  19. Workflow engine overloadDeterministic steps run on a workflow engine with 6 slots (200 ms each) between the agent’s decisions.MediumSolve
  20. Loading skills every stepThe agent loads skills (instructions and scripts) from a registry with 4 slots on every step.EasySolve
  21. Agent jobs run twiceAgent jobs are queued and run by workers. When an acknowledgement is lost, the queue redelivers and the job’s side effect (sending an email) happens twice.HardSolve
  22. Tracing every agent stepEach step writes a trace span to an observability API capped at 1,200 a minute; agents make about 30 steps a second.EasySolve

Multi-agent and harness 12

  1. Subagents swamp the search toolA lead agent fans each question out to 8 parallel subagents, and each subagent searches the web. The search tool has 8 slots.MediumSolve
  2. Multi-agent token billEvery subagent sends 3,000 tokens of context. Eight subagents per question multiply token use past the provider quota.HardSolve
  3. A coding harness that never stopsA long-running coding harness takes 20 turns per task, and a 1% tool failure rate sinks a fifth of tasks.EasySolve
  4. Harness compaction backs upThe harness compacts context on every turn with a summariser that has 4 slots.MediumSolve
  5. Harness progress notesThe harness writes progress notes and a checkpoint after every turn to a store with 6 slots.MediumSolve
  6. Harness test runsA coding harness runs the test suite in a sandbox every turn. 8 sandboxes, about 1.2 s per run.HardSolve
  7. Harness loses its modelA long-running harness depends on one model provider, which is down for 20 seconds mid-run.MediumSolve
  8. The planner model is the bottleneckA planner model plans each step for worker agents, but its deployment allows only 20 streams.MediumSolve
  9. The lead agent over-plansA lead agent re-plans 10 times per task before delegating, and each plan calls a flaky tool.MediumSolve
  10. Fragile subagentsEight subagents each call a tool that fails 1% of the time; any subagent failure fails the question.HardSolve
  11. Research agents share few browsersA deep-research system fans out to 5 subagents that each browse the web; the browser pool has 16 browsers.HardSolve
  12. Harness traces over quotaEvery harness turn is traced to an API capped at 600 spans a minute.EasySolve

AI security and governance 14

  1. Prompt injection on a support botAt 8 s, attackers send 400 messages a second with hidden instructions (“ignore previous instructions…”). Nothing screens them.MediumSolve
  2. Injected instructions in emailAn email agent reads inbound messages and can call tools. Attackers send 200 emails a second containing instructions to forward data.HardSolve
  3. Poisoned web pages in RAGA web-search assistant retrieves pages that contain hidden instructions. 300 poisoned requests a second arrive alongside real users.HardSolve
  4. Malicious issue textA coding harness reads issue descriptions. Attackers file issues with instructions to exfiltrate secrets, 100 a second.HardSolve
  5. Injection filter blocks real usersA strict prompt-injection filter flags 4% of real users as attackers.MediumSolve
  6. Guardrails too strictInput guardrails block 3% of legitimate questions as policy violations.MediumSolve
  7. DDoS on an LLM APIAt 8 s a 20,000 req/s flood hits your public LLM API. Every flood request costs GPU time.MediumSolve
  8. Bots abuse the free tierBots farm free-tier accounts and send 2,000 requests a second to your model.EasySolve
  9. Spend governor set too lowA spend governor caps model calls at 1,200 a minute to protect the budget, but legitimate traffic is 30 a second.EasySolve
  10. Credential stuffing on an AI appAttackers replay leaked passwords against your AI app’s login at 3,000 a second.MediumSolve
  11. Redacting every agent stepEvery agent step passes through a PII redactor that handles 2 calls at a time, about 40 ms each.MediumSolve
  12. Policy service on the chat pathEvery chat checks a data-access policy service with 8 slots and 40 ms per check.EasySolve
  13. Audit logging every model callEvery model call is written to an audit log (SIEM) that accepts 30 writes in flight.MediumSolve
  14. Two attacks at onceA DDoS flood and a prompt-injection campaign arrive together.HardSolve

AI products 16

  1. Support bot launch dayA customer-support bot with a semantic cache, retrieval and a hosted model launches to 60 questions a second. The token quota wasn’t raised.MediumSolve
  2. IDE code completionsInline completions: 400 requests a second, short outputs, and only 64 streams on the completion model.MediumSolve
  3. Meeting summaries time outSummaries are 1,000 tokens at 80 tokens a second; the app times out after 10 seconds.EasySolve
  4. AI search engine retrievalAn AI search engine rewrites the query, runs hybrid search and answers. Hybrid search has 30 slots at 1,200 queries a second.MediumSolve
  5. Translation with per-language adaptersA translation product serves 40 language adapters from one multi-LoRA server with 24 slots.MediumSolve
  6. Content moderation queueEvery post is checked by a moderation model with 30 slots, about 600 ms per post, at 80 posts a second.MediumSolve
  7. Invoice extractionInvoices are OCR’d and extracted on upload: 20 OCR slots, 1.2 s per page, 25 pages a second.EasySolve
  8. Voice replies lagA voice assistant’s text-to-speech has 20 slots and takes about 600 ms per reply at 50 replies a second.MediumSolve
  9. Shopping agent: payments downA shopping agent checks out through an external payment API that is down from 15 s to 30 s.HardSolve
  10. Travel agent: airline partner overloadedA travel agent books flights through an airline’s agent over A2A, which handles 15 tasks at once.MediumSolve
  11. Data analyst agent and the warehouseA data-analyst agent writes SQL against a warehouse that runs 8 queries at once, about 800 ms each.MediumSolve
  12. SRE agent runs too longAn on-call SRE agent investigates incidents in 15 steps of model calls and tool queries.MediumSolve
  13. Medical scribe: redaction bottleneckA medical scribe transcribes visits, redacts PII and drafts notes. The redactor has 1 vCPU and 20 ms per note, at 100 notes a second.HardSolve
  14. Legal research with GraphRAGLegal questions traverse a citation graph: 12 ms of CPU each on 4 vCPUs, at 500 questions a second.HardSolve
  15. Sales agent reads hostile repliesA sales agent reads inbound replies and updates the CRM. A campaign of replies with injected instructions arrives at 150 a second.HardSolve
  16. Video generation farmText-to-video takes about 20 s per clip on 40 GPUs; users request 3 clips a second.MediumSolve

AI systems 11

  1. RAG support assistant40 questions/s, 4,000-token prompts, a 6M tokens/min quota and a 25% answer-cache hit rate.MediumSolve
  2. Agent runaway15 tasks/s. The agent makes 8 model and tool calls per task; the model allows 4,000 calls a minute and 120 concurrent streams.HardSolve
  3. LLM gateway with fallback40 chats/s through your AI gateway and a prompt cache to one hosted model. At 20 s the provider has a 15 s outage.MediumSolve
  4. MCP agent platform4 tasks/s. The agent thinks (model) and acts (tools through an MCP gateway) 6 times per task, with vector memory and results in S3.HardSolve
  5. Document Q&A on S3 and a vector DB40 req/s: 90% questions, 10% uploads. Uploads go to S3 and a queue; workers chunk and embed them into a vector DB. Questions use the same embedding model.HardSolve
  6. RAG at twice the trafficYour support assistant doubles to 80 questions a second. The model’s token quota does not.MediumSolve
  7. Semantic cache that never hitsThe semantic cache’s similarity threshold is so strict it hits 5% of the time. Nearly every question reaches the model.EasySolve
  8. The agent that thinks too muchAn agent takes 12 model-and-tool steps per task. Steps multiply load, cost and failures.MediumSolve
  9. Model provider outageYour chat product depends on one model provider. At 20 s it is down for 15 seconds.MediumSolve
  10. Ten steps of MCP toolsAgents call MCP tools 10 times per task. A 1% tool error rate becomes a 10% task failure rate.HardSolve
  11. Uploads starve questionsQuestions and document ingestion share one embedding quota. Uploads double and both paths start failing.HardSolve