# Evaluate your RAG system

To know whether your RAG system answers well, you measure two things separately: did retrieval fetch the right passages, and did the model answer faithfully from them. You build a small labelled question set, score each stage against it, and re-run those scores on every change. Vibes are not evaluation.

If you are still assembling the pipeline, read [RAG fundamentals](/guides/rag-fundamentals) first. This guide is about proving it works.

## Why do you score retrieval and generation separately?

A RAG answer can be wrong for two different reasons, and they need different fixes:

- **Retrieval missed.** The passage that holds the answer never reached the prompt, so the model had no way to produce it. The fix is in [chunking](/guides/chunk-your-documents), [embeddings](/guides/embeddings-for-beginners), or search.
- **Generation drifted.** The right passage was in the context, but the model ignored it, contradicted it, or padded the answer with invention. The fix is in the prompt or the grounding instructions.

These look identical from the outside: both produce a confident wrong answer. If you only score the final answer, you cannot tell which half to repair, so you guess. Scoring the two stages on their own turns guessing into a decision.

## Step one: build a golden set

You cannot measure anything without a reference. The reference is a golden set: a list of real questions, the correct answer for each, and the source passage that answer should come from.

Start with thirty to fifty questions. That is enough to catch a regression and small enough to label by hand in an afternoon. Cover the spread of how people ask:

- **Plain lookups.** "What is the refund window." One passage holds the answer.
- **Multi-passage questions.** "Compare the starter and pro plans." The answer needs two or three passages.
- **Exact-term questions.** "What does error E-4021 mean." Tests whether search finds literal strings, not just meaning.
- **Unanswerable questions.** "What is the CEO's home address." The corpus does not contain this. The correct behaviour is to decline, and you must test for it.

Store the set as plain data so you can version it next to your code. A row per question is enough:

```json
{
  "id": "q-014",
  "question": "How long is the refund window?",
  "expected_answer": "30 days from the delivery date.",
  "source_docs": ["docs/billing/refunds.md"],
  "answerable": true
}
```

Claude can draft this set for you. Point the [Claude Code CLI](/guides/claude-code-quickstart) at your real documents and ask it to generate candidate questions and answers, then you check them. Drafting is fast; the value is in your verification, so do not skip the read-through.A question drafted straight from a passage tends to echo that passage's exact wording, so retrieval finds it easily and your recall looks great while real users phrasing things differently get worse results. During the read-through, reword each question into how a customer would ask, with their own synonyms, so the set tests retrieval instead of rewarding string overlap.

```text
Read every file under docs/billing/ and docs/shipping/. Generate 30 question-and-answer pairs a real customer would ask, covering plain lookups, comparisons, and at least 5 questions the docs do NOT answer. For each, record the exact file the answer comes from, or null if unanswerable. Output as a JSON array matching the schema in eval/golden-set.json. Do not invent answers that are not in the files.
```

## Step two: score retrieval on its own

Run each golden question through retrieval only and compare the passages it returns against the `source_docs` you labelled. Three numbers tell you most of what you need.

| Metric | Question it answers | When it matters most |
| --- | --- | --- |
| Recall@k | Of the passages that should be found, how many landed in the top k | The headline number. A miss here caps the whole system |
| Precision@k | Of the k passages returned, how many were relevant | High recall with low precision floods the prompt with noise |
| MRR | How early did the first correct passage appear | When the model reads the top of the context best |

Recall@k is the one to watch. If the answer is not in the retrieved context, no amount of prompt tuning will recover it, so a retrieval miss is a hard ceiling on answer quality. Track it first.

Precision@k is the counterweight. You can always raise recall by returning more passages, but each junk passage costs tokens and gives the model more to be distracted by. You want both high, and `k` set so recall is strong without burying the good passage.

MRR rewards ordering. A correct passage at position one is worth more than the same passage at position eight, because models attend best to the start and end of a long context. If MRR is low while recall is fine, a reranker is the usual fix.

### How do recall and precision trade off?

The two move against each other as you change `k`, the number of passages you return. Raise `k` and recall climbs, because more passages means a higher chance the right one is included. But precision falls, because each extra passage is more likely to be junk. The job is to find the `k` where recall is already high and precision has not yet collapsed.

A worked example makes the trade concrete. Say a question's answer lives in one passage:

- At `k=3`, you return three passages, the right one is among them, and two are noise. Recall is high, precision is one in three.
- At `k=10`, recall is still high but precision drops to one in ten, and the model now reads nine irrelevant passages looking for the one that matters.

Sweep `k` across your golden set and pick the smallest value where recall plateaus. Returning more after that point buys nothing and costs tokens.

## Step three: score the generated answer

Once retrieval is solid, score what the model does with it. The four metrics popularised by the Ragas evaluation framework split cleanly across the two stages:

| Metric | What it checks | Stage |
| --- | --- | --- |
| Context recall | Did retrieval fetch all the supporting facts | Retrieval |
| Context precision | Was the retrieved context mostly relevant | Retrieval |
| Faithfulness | Is every claim in the answer backed by the context | Generation |
| Answer relevancy | Does the answer address the question | Generation |

Faithfulness is your guard against confident invention. A faithful answer makes no claim the context does not support. When faithfulness drops while context recall stays high, the bug is in generation: the model has the facts and is adding to them anyway. Tighten the grounding instruction.

Answer relevancy catches the opposite problem: an answer that is technically grounded but dodges the question or pads it with filler.

### The four metrics as a diagnosis grid

Read the two retrieval metrics and the two generation metrics together and they point straight at the broken stage:

| Context recall | Faithfulness | Diagnosis |
| --- | --- | --- |
| High | High | Working. Lock in the baseline |
| Low | High | Retrieval miss. Fix chunking or search |
| High | Low | Model invents on good context. Fix grounding |
| Low | Low | Both stages need work. Start with retrieval |

Always fix retrieval before generation. A generation metric measured on top of bad context is noise: the model cannot be faithful to facts it never received, so the faithfulness number means nothing until recall is solid.

Scoring faithfulness by hand does not scale. The common approach is an [LLM-as-judge](/guides/stop-claude-making-things-up): you give a model the question, the retrieved context, and the answer, and ask it to rule on whether each claim is supported. Treat the judge as an instrument you also calibrate. Spot-check its rulings against your own on a handful of cases before you trust the aggregate.The trap is treating the judge's faithfulness score as ground truth and tuning your whole pipeline to lift it, without ever checking whether the judge agrees with a human. An uncalibrated judge has its own blind spots, so you can spend a week chasing a number that does not track real answer quality, then ship a system your users find worse. Hand-label twenty answers, measure how often the judge matches you, and fix the judge prompt until that agreement is high before you trust a single aggregate score.

## Running it through the CLI

You do not need a separate evaluation service to start. Drive the loop with the Claude Code CLI: it reads your golden set, runs each question, and writes a scored report you can diff against the last run.

A slash command keeps the routine identical between runs. Save the instructions once and invoke them by name:

```markdown
---
description: Score the RAG pipeline against the golden set
---

Read eval/golden-set.json. For each question:
1. Retrieve passages and record which source_docs appeared and at what rank.
2. Generate an answer from only those passages.
3. Judge faithfulness and answer relevancy against the context.

Compute recall@k, precision@k, MRR, and average faithfulness.
Write results to eval/runs/.json and print a table comparing
this run to the previous file in eval/runs/.
```

Then each run is one line:

```bash
claude /eval-rag
```

Because the routine lives in a file, every teammate runs the same checks the same way, and the diff between two run files is your regression signal. To wire the same routine into a richer toolchain, see [MCP fundamentals](/guides/mcp-fundamentals) for connecting evaluation data sources, and [hooks and guardrails](/guides/hooks-and-guardrails) to fail a commit when a score drops.

## How do you catch RAG regressions automatically?

The point of a saved score is to compare against it. Run the evaluation on a schedule and on every change to retrieval, chunking, the prompt, or the index, then alert when a number falls.

A pre-commit or CI hook can run the suite and block a change that drops faithfulness or recall below a floor you set:

```json
{
  "hooks": {
    "pre-commit": "claude /eval-rag && node eval/check-thresholds.js"
  }
}
```

The threshold check reads the newest run file and exits non-zero if, for example, recall@5 is under 0.85 or faithfulness is under 0.90. Pick the floors from your own baseline, not from a blog post: run the suite once on the system you have today and set the floor a little under each current number.

## What should you track over time?

A single run is a snapshot. The signal is in the trend, because the thing that breaks RAG in production is not a bad launch, it is quiet drift as the corpus changes.

- **Retrieval recall@k.** The leading indicator. A slow decline means your index has drifted from the live source, or new content is chunked badly. Re-index and re-check.
- **Faithfulness.** A drop here without a retrieval drop points at a prompt change or a model swap that loosened grounding.
- **Unanswerable accuracy.** The share of unanswerable questions the system correctly declines. This one degrades silently and is where confident invention hides.
- **Latency per stage.** Retrieval time and generation time, tracked apart. A reranker that lifts MRR but doubles latency is a trade you want to see, not discover in a complaint.
- **Cost per query.** Returning more passages raises recall and tokens together. Watch them as a pair so a recall gain does not quietly become a budget problem. See [token usage and cost](/guides/token-usage-and-cost).

## A worked loop

Put it together and the workflow is short:

1. Build the golden set, thirty to fifty labelled questions including unanswerable ones.
2. Run retrieval alone. Read recall@k first. If it is low, fix chunking, embeddings, or [hybrid search](/guides/hybrid-search-for-rag) before touching anything else.
3. Once retrieval holds, run the full pipeline and score faithfulness and answer relevancy with an LLM judge you have spot-checked.
4. Save the run, set thresholds a little under today's numbers, and wire the suite into a hook.
5. Re-run on every change and on a schedule. Investigate any number that falls.

Evaluation is not a phase you finish. It is the instrument that tells you whether each change helped, and the only honest answer to "is the RAG system good." Next, read [RAG for coding agents](/guides/rag-for-coding-agents), where the standard recipe needs adjusting.