# Make your RAG answers accurate

A wrong RAG answer is almost always a retrieval problem, not a model problem. If
the right passage never reaches Claude, no prompt can save the answer. So you fix
accuracy at the retrieval stage: cut better chunks, send sharper queries, and add
metadata filters that narrow the search before it runs.

This guide assumes you already know the pipeline. If you do not, read
[RAG Fundamentals](/guides/rag-fundamentals) first, then come back here for the
quality work.

## Diagnose before you tune

Do not change anything until you know which stage failed. A bad answer has one of
three causes, and they need different fixes.

The check is mechanical. Take a question that gets a wrong answer, look at the
chunks your system retrieved, and ask: was the correct passage in that
set at all? If it was not, no amount of prompt tuning helps. If it was there but
cut in half, your chunking split the fact. If the full passage was present and
the answer is still wrong, only then is it a prompting problem.

Claude is good at this triage. Paste a failing question, the retrieved chunks,
and the wrong answer, and ask it to locate the break.

Here is a RAG failure. The question, the chunks my retriever returned, and the answer my system gave are below. Tell me one thing: was the information needed to answer correctly present in the retrieved chunks? If yes, the bug is in my prompt. If no, the bug is in retrieval. Quote the evidence for your verdict.

## How should I chunk documents for RAG?

Chunking sets the ceiling on everything downstream. A chunk that splits a fact in
half cannot be retrieved whole, no matter how good your search is.

| Strategy | How it splits | Use when |
| --- | --- | --- |
| Fixed-size | Every N characters | A quick baseline only |
| Recursive | Paragraph, then sentence, then word | General prose, the common default |
| Semantic | At topic shifts | Mixed documents, higher cost |
| Structural | At headings, list items, code blocks | Docs, wikis, structured files |

Three rules hold across strategies:

- **Respect structure.** Split at headings, paragraphs, and list boundaries
  before you split mid-sentence. A heading plus its body is a self-contained
  unit; keep them together.
- **Add overlap.** Carry roughly ten to twenty percent of the previous chunk into
  the next one, so a fact sitting on a boundary survives in at least one chunk.
- **Keep chunks self-contained.** A chunk that says "it grew by 3 percent" is
  useless without knowing what "it" is. This is the core problem
  contextual retrieval solves.

### Add context to each chunk

A raw chunk often loses the document it came from. Anthropic's contextual
retrieval fixes this by prepending a short, model-written sentence that situates
the chunk in its source before you embed it. A chunk reading "The company's
revenue grew by 3% over the previous quarter" becomes "This chunk is from an SEC
filing on ACME corp's Q2 2023 performance; the prior quarter's revenue was $314
million. The company's revenue grew by 3% over the previous quarter."

The generated context is short, around fifty to one hundred tokens, produced with
a prompt that asks the model to situate the chunk within the whole document.
Anthropic reports this cut the top-20 retrieval failure rate by 35 percent on its
own, and by 49 percent when paired with keyword search. Running this over a large
corpus calls the model once per chunk, so use
prompt caching
to avoid re-sending the full document for every chunk it contains.

You can generate the augmented chunks with the Claude Code CLI as a one-off
build step over your source files.

Read every Markdown file under docs/. For each file, split it into chunks at heading and paragraph boundaries, keeping each chunk under 800 tokens with about 15 percent overlap. Before each chunk, prepend one or two sentences that explain where this chunk sits in the document and what it is about, using the file title and surrounding sections. Write the result to build/chunks.jsonl, one chunk per line with fields: source_path, heading, context, text.

## What is hybrid search and why do I need it?

Pure vector search captures meaning but misses exact strings. A user searching
for error code "TS-999" or the function `validateSession` wants a literal match,
and semantic similarity can rank a vaguely related passage above the exact one.

The fix is hybrid search:
run a keyword search and a vector search, then merge the two ranked lists.

- **Keyword search** uses BM25,
  which scores documents by exact word matches weighted by rarity. It nails IDs,
  error codes, and function names.
- **Vector search** scores by meaning, so "how do I sign in" can match a passage
  titled "authentication flow" that shares no words.Embed the query with the exact same model and version you indexed the documents with. Swap one side and the similarity scores turn to noise, so re-embed the whole corpus whenever you change the model, never just the new chunks.
- **Rank fusion** merges both lists into one. Anthropic found combining
  contextual embeddings with contextual BM25 reduced the top-20 retrieval failure
  rate by 49 percent versus embeddings alone.

If you store your data in Postgres, you already have both halves: a full-text
index for the keyword side and the `pgvector` extension for the vector side. Many
dedicated vector stores expose hybrid search as a single query option. For the
full treatment of fusing the two, see [hybrid search: keywords plus embeddings](/guides/hybrid-search-for-rag).

## How does metadata filtering improve RAG accuracy?

The cheapest accuracy gain is to search less. If a question only concerns the
2026 pricing docs, searching the 2024 archive can only add noise. Attach metadata
to every chunk at index time, then filter on it at query time so the search runs
over the right subset.

Useful metadata fields:

- **Source and section.** The file path and heading the chunk came from, so you
  can scope a search to one product area.
- **Date.** When the content was written or last updated, so you can prefer fresh
  passages and exclude stale ones.
- **Type.** Whether the chunk is reference, a tutorial, an API description, or a
  changelog, so a "how do I" question can favour tutorials.
- **Access.** Which users or roles may see the chunk, so retrieval never returns
  content the asker is not allowed to read.

Store these alongside each chunk so a query can request, for example, only chunks
where `type = "tutorial"` and `date >= "2026-01-01"`. A
pre-filter
narrows the candidate set first, which both speeds up the search and removes whole
classes of wrong answers.

The metadata is only as good as the question's ability to use it. That is the
next problem.

## Sharpen the query

The user's words are rarely the best search query. They use pronouns, they ask
two things at once, and they leave the topic implicit because they know what they
mean. Three rewrites help.

### Rewrite for retrieval

Turn a conversational question into a search-shaped one before you embed it.
"What about the second one?" cannot be searched; "What is the refund window for
the Pro plan?" can. Use the conversation so far to resolve pronouns and restore
the dropped subject.

Rewrite the user's latest message into a standalone search query. Resolve every pronoun using the conversation above, spell out any implied subject, and keep it to one clear question. Return only the rewritten query, nothing else.

### Split multi-part questions

"What does the API return and how do I authenticate?" is two searches wearing one
question. Retrieving for the merged query finds passages that are mediocre on both
topics and excellent on neither. Split it, retrieve for each part, and combine the
results.

### Expand sparse queries

A two-word query gives the embedding little to work with. Generating a few
alternative phrasings and searching for all of them widens recall. This is one
form of query expansion;
keep the variants close to the original intent so you widen recall without
dragging in unrelated topics.

## What does a reranker do in RAG?

Your first-pass search is built for speed over a large index, so it ranks
approximately. A reranker
reads the query and each candidate passage together and re-scores them, which is
slower per item but far more accurate. So you run it on a shortlist, not the whole
index.

The pattern is [fetch wide, then rank narrow](/guides/rerank-your-rag-results). Anthropic's tests retrieved the top
150 chunks, reranked them, and kept the top 20 for the model. Adding reranking on
top of contextual hybrid search took the total top-20 failure reduction to 67
percent, from a 5.7 percent failure rate down to 1.9 percent. Reranking adds a
runtime step, so it costs latency and money; rank fewer candidates if that
budget is tight, more if accuracy matters more than speed.

## How many chunks should I send to the model?

Sending more chunks is not safer. Two effects push back.

- **Lost in the middle.** Models attend best to the start and end of a long
  context and worst to the middle. Burying the key passage in the middle of forty
  chunks hides it. Send fewer, stronger chunks, and place the best one first or
  last.
- **Distraction.** Every irrelevant chunk is a chance for the model to anchor on
  the wrong fact. Precision in the final set matters as much as recall.

Tune the count to your task. Anthropic found top-20 best for its benchmark after
reranking, but a focused question over clean docs can do better on five. Measure
it; do not guess.

## Measure retrieval on its own

You cannot improve what you do not measure, and a bad retriever feeding a good
model looks exactly like a bad model. Score the retrieval stage separately.

- **Recall** asks whether the right chunks were fetched at all. This is the
  high-signal metric: if the answer is not in the retrieved set, the model cannot
  produce it.The tempting shortcut is to grade only the final answer end to end and call that your accuracy number, but that mixes retrieval and prompting into one figure, so a hidden recall drop hides behind a forgiving model until it suddenly fails on a hard question. Score recall against a fixed question-to-chunk set on its own, separately from answer quality, so you can see which stage moved before and after every change.
- **Precision** asks how many fetched chunks were relevant. Low precision
  means noise that distracts the model.

Build a small evaluation set: real questions paired with the chunk that should
answer each one. After every change to chunking, search, or filtering, re-run it
and watch recall. Claude can help you build the set from your own documents.

Read the chunks in build/chunks.jsonl. Generate 25 evaluation questions a real user might ask, each one answerable from exactly one chunk. For each, record the question and the source_path plus heading of the chunk that answers it. Write the result to eval/retrieval-set.jsonl. Make the questions varied: some exact-term lookups, some conceptual, some multi-part.

## A working order

When accuracy is low, change one thing at a time and re-measure. Doing
everything at once tells you nothing about what worked.

1. Confirm it is a retrieval problem, not a prompt problem.
2. Fix chunking: respect structure, add overlap, make chunks self-contained.
3. Add metadata and pre-filter so each search runs over the right subset.
4. Add hybrid search so exact terms and meaning both count.
5. Rewrite and split queries before they hit the index.
6. Add a reranker over a wide shortlist.
7. Tune how many chunks reach the model, watching for lost-in-the-middle.

Each step is independent, and each is measurable against your evaluation set. Get
chunking and retrieval right and the model has what it needs; the rest is
grounding, which [RAG Fundamentals](/guides/rag-fundamentals) covers. For the
context window itself, see
[Context Engineering](/guides/context-engineering).