AI systems are not magic. Under the surface they are compositions of a small number of repeating ideas. Once you can name these patterns you stop re-inventing them and start choosing between them deliberately. This guide covers the nine that show up constantly, with a plain description of each, when it earns its place, and the real cost you pay for using it.
What are the core AI engineering patterns?
There are nine. Use this table as a fast reference. Details and diagrams follow below.
| Pattern | One-line what | Reach for it when | The trade |
|---|---|---|---|
| Retrieval-Augmented Generation | Inject retrieved documents into context before generation | The model needs facts it was not trained on | Retrieval latency + chunking quality determines answer quality |
| Tool Use | Model calls external functions and uses results | The model needs live data or actions | Each round trip adds latency and a potential failure point |
| The Agent Loop | Model plans, acts, observes, repeats | The task cannot be solved in one shot | Loops can run away; set a hard step limit |
| Orchestrator and Subagents | One coordinator hands work to specialist agents | The task is wide, parallelisable, and high-value | Roughly 4x token cost per subagent vs a single call |
| Routing | A classifier sends a query to one of several handlers | You have meaningfully different query types | The router itself can misclassify |
| Verifier / Critic | A second model checks the first model's output | Correctness matters more than speed | Doubles inference cost at minimum |
| Structured Output | Force the model to emit a schema-valid object | Downstream code needs to parse the response | Strict schemas can degrade generation quality on edge cases |
| Prompt Caching | Reuse a processed prefix across requests | You repeat the same large context | Cache misses still pay full price; TTL is short by default |
| Evals | Systematic measurement of output quality | You are shipping, iterating, or debugging a system | Requires investment upfront to define what "good" means |
What is Retrieval-Augmented Generation (RAG)?
RAG gives the model facts it was never trained on by fetching them at query time. You index your documents into a vector store, embed the user's query, retrieve the closest chunks, and prepend them to the prompt.
When to use it. Your data changes faster than you can retrain, the model hallucinates on domain-specific questions, or the answer lives in a private corpus.
The trade. Answer quality is capped by retrieval quality. A chunk that almost matches the query is worse than no chunk at all because it confidently misleads the model. You need to tune chunk size, overlap, and embedding model before the system is trustworthy. Retrieval also adds latency on every request.Common gotchaWhen a RAG answer is wrong, do not blame the model first.Log the retrieved chunks and check them by hand: most bad answers are a retrieval miss, not a generation failure, and you cannot fix what you are not measuring separately.
Agentic RAG is the next step: instead of a fixed retrieve-then-generate flow, the model decides when to retrieve and issues multiple queries mid-reasoning. It is more powerful but harder to debug.
See also: RAG Fundamentals and RAG for Coding Agents.
Tool Use
The model emits a structured call that your runtime intercepts, executes, and returns as an observation. The model sees the result and decides what to do next. Common tools: web search, a code interpreter, a database query, a file reader.
When to use it. The model needs information that changes (stock prices, search results, your database) or needs to take an action in the world (send an email, run code, write a file).
The trade. Every tool call is a round trip. Latency compounds if the model chains several calls. The tool can also fail, return unexpected data, or return correct data the model misinterprets. You need error handling at the tool layer.
See also: Tool Use and Function Calling and MCP Fundamentals.
What is an agent loop?
An agent loop is a model that does not answer once and stop. It plans a step, takes an action (usually a tool call), reads the result, updates its plan, and repeats until it decides the task is complete.
When to use it. A task requires multiple dependent steps where you cannot know the exact sequence upfront. Writing a test suite, debugging a failing CI build, researching a topic across multiple sources.
The trade. Loops can stall or spiral. Without a hard step ceiling the model can spend indefinitely. Set a max-iteration limit, watch your token usage, and add a timeout. A loop that never terminates is a bug you pay for in both money and wall-clock time.
Orchestrator and Subagents
One orchestrator model decomposes a task, hands pieces to subagents, and assembles the results. Subagents can run in parallel if their work is independent.
When to use it. The task is wide: researching a dozen companies, generating and reviewing a large codebase, running parallel test strategies. The subtasks must be meaningfully independent. If they share state or depend tightly on each other, a single agent in a loop is cleaner.
The trade. Anthropic's guidance puts the token cost of a multi-agent system at roughly fifteen times a plain chat call. Only reach for this pattern when the value of the task justifies that cost. Orchestration also adds failure modes: a subagent that returns bad output can corrupt the final result silently.
See also: How Agents Fan Out and Multi-Agent Architectures.
Routing
A lightweight classifier reads the incoming query and sends it to one of several specialised handlers. Handlers can be different prompts, different models, different tools, or entirely different pipelines.
When to use it. Your system serves meaningfully different query types that benefit from different treatment. A support bot that handles both technical debugging and subscription questions. A coding assistant that separates explanation requests from code-generation requests.
The trade. The router can misclassify, and misclassification sends the query to a handler with the wrong instructions, the wrong context, or the wrong model. You need evals on the router itself, not only the downstream handlers. Keep the routing taxonomy small. Three to five categories is usually safer than ten.
Verifier and Critic
A second model (or a second call to the same model with a different prompt) reads the first model's output and either approves it, flags errors, or rewrites it.
When to use it. Correctness matters more than speed. Legal summaries, code that ships to production, medical triage, financial analysis. Also useful as a quality gate before expensive downstream work.
The trade. You double the inference cost at minimum. If the critic and the generator share the same model family, they share the same failure modes, so the critic catches surface errors but misses systematic ones. Use a different model or a different prompt framing for the critic to get independent signal.
A useful variant: the critic does not rewrite, it only scores. The system retries generation until the critic score exceeds a threshold. This is effective but can loop indefinitely on a hard task.
Structured Output
You constrain the model to emit a JSON object (or another schema) rather than free prose. Your code then parses the object without needing to extract data from natural language.
When to use it. A downstream function needs to consume the model's answer programmatically. Extracting fields from a document, classifying intent, filling a database record, driving a UI component.
The trade. Strict schemas can degrade generation quality on edge cases. The model sometimes forces an answer into a field where the honest answer is "unknown." If the schema does not have a nullable or "unclear" field, you get confident but wrong data. Design your schemas with uncertainty in mind. Require only the fields you need.
See also: Structured Outputs.
What is prompt caching?
Prompt caching reuses a processed prefix across requests. You mark a large, stable prefix (a system prompt, a long document, a reference codebase) as a cache breakpoint. The provider stores the processed state of everything up to that point. Subsequent requests that share the same prefix read from the cache instead of reprocessing it, paying roughly 10% of the normal input token price.
When to use it. Your system prompt or context is long and repeats across many requests: a coding assistant with a large rules file, a customer support bot with a full product manual, a legal tool with a lengthy case document. Claude Code uses this automatically on your CLAUDE.md and open files.
The trade. The default cache TTL is five minutes, reset on each cache hit. An optional one-hour TTL is available at higher write cost. Cache misses still pay the full input rate, so low-traffic systems may never break even. The prefix must be byte-for-byte identical for a cache hit to occur, so any variation (dynamic timestamps, personalised greetings at the top) busts the cache.
See also: Prompt Caching.
Evals
Evals are systematic measurements of output quality across a representative set of inputs. You define what "good" means, collect examples, and run the suite every time you change the prompt, model, or system design.
When to use it. Always, once you are past a prototype. Evals are the difference between "it felt better in my tests" and "it is demonstrably better on the cases that matter." They are how you detect regressions when you update your system prompt, swap models, or change retrieval logic.
The trade. Evals require upfront investment. You need to define the evaluation criteria, label examples, and decide whether to use model-graded scoring, exact-match checks, or human review. Model-graded evals (where a second model scores the output) are the most scalable but inherit the biases of the grader. Start small: ten to fifty representative cases beats a thousand random ones.
See also: Build evals for a production AI system.
How do AI engineering patterns work together?
Most production AI systems use several of these patterns together. A common combination: a router classifies the query, the chosen handler uses RAG to fetch context, the model runs a short agent loop with tool calls, and structured output formats the final result. A critic gate sits before any answer goes to the user.
The table at the top of this guide is designed for scanning when you are deciding which patterns belong in a new system. Start with the simplest combination that could work. Add patterns only when a specific gap makes them worth the added cost and complexity.Common gotchaThe common mistake is reaching for the heavy patterns first: a multi-agent orchestrator, a critic gate, and an agent loop on day one, before you have a single eval.Without evals you cannot tell whether any of that machinery helped or just burned tokens, so you keep stacking patterns on a system you cannot measure. Build the eval set first, ship the plainest single call that passes it, and add a pattern only when the numbers say the simpler version falls short.
Quick-reference prompt starters
When you want Claude Code to help you apply one of these patterns, a well-framed prompt gets you there faster.
Bookmark this page and come back to it when a new project demands a design decision. The patterns do not change. The models do.