Make your RAG answers accurate

A wrong RAG answer is almost always a retrieval problem, not a model problem. If the right passage never reaches Claude, no prompt can save the answer. So you fix accuracy at the retrieval stage: cut better chunks, send sharper queries, and add metadata filters that narrow the search before it runs.

This guide assumes you already know the pipeline. If you do not, read RAG Fundamentals first, then come back here for the quality work.

Diagnose before you tune

Do not change anything until you know which stage failed. A bad answer has one of three causes, and they need different fixes.

The check is mechanical. Take a question that gets a wrong answer, look at the chunks your system retrieved, and ask: was the correct passage in that set at all? If it was not, no amount of prompt tuning helps. If it was there but cut in half, your chunking split the fact. If the full passage was present and the answer is still wrong, only then is it a prompting problem.

Claude is good at this triage. Paste a failing question, the retrieved chunks, and the wrong answer, and ask it to locate the break.

How should I chunk documents for RAG?

Chunking sets the ceiling on everything downstream. A chunk that splits a fact in half cannot be retrieved whole, no matter how good your search is.

Three rules hold across strategies:

• Respect structure. Split at headings, paragraphs, and list boundaries before you split mid-sentence. A heading plus its body is a self-contained unit; keep them together.
• Add overlap. Carry roughly ten to twenty percent of the previous chunk into the next one, so a fact sitting on a boundary survives in at least one chunk.
• Keep chunks self-contained. A chunk that says "it grew by 3 percent" is useless without knowing what "it" is. This is the core problem contextual retrievalA technique from Anthropic where a short, model-written sentence of context is prepended to each chunk before embedding, so the chunk explains itself. solves.

Add context to each chunk

A raw chunk often loses the document it came from. Anthropic's contextual retrieval fixes this by prepending a short, model-written sentence that situates the chunk in its source before you embed it. A chunk reading "The company's revenue grew by 3% over the previous quarter" becomes "This chunk is from an SEC filing on ACME corp's Q2 2023 performance; the prior quarter's revenue was $314 million. The company's revenue grew by 3% over the previous quarter."

The generated context is short, around fifty to one hundred tokens, produced with a prompt that asks the model to situate the chunk within the whole document. Anthropic reports this cut the top-20 retrieval failure rate by 35 percent on its own, and by 49 percent when paired with keyword search. Running this over a large corpus calls the model once per chunk, so use prompt cachingReusing the cached document portion of a prompt across many calls so you do not re-send and re-process the same tokens each time. to avoid re-sending the full document for every chunk it contains.

You can generate the augmented chunks with the Claude Code CLI as a one-off build step over your source files.

What is hybrid search and why do I need it?

Pure vector search captures meaning but misses exact strings. A user searching for error code "TS-999" or the function validateSession wants a literal match, and semantic similarity can rank a vaguely related passage above the exact one.

The fix is hybrid searchRunning keyword search and vector search together, then merging their ranked results into one list.: run a keyword search and a vector search, then merge the two ranked lists.

• Keyword search uses BM25A classic keyword-ranking algorithm that scores documents by exact word matches weighted by how rare each word is. Strong on exact terms, blind to meaning., which scores documents by exact word matches weighted by rarity. It nails IDs, error codes, and function names.
• Vector search scores by meaning, so "how do I sign in" can match a passage titled "authentication flow" that shares no words.Common gotchaEmbed the query with the exact same model and version you indexed the documents with.Swap one side and the similarity scores turn to noise, so re-embed the whole corpus whenever you change the model, never just the new chunks.
• Rank fusion merges both lists into one. Anthropic found combining contextual embeddings with contextual BM25 reduced the top-20 retrieval failure rate by 49 percent versus embeddings alone.

If you store your data in Postgres, you already have both halves: a full-text index for the keyword side and the pgvector extension for the vector side. Many dedicated vector stores expose hybrid search as a single query option. For the full treatment of fusing the two, see hybrid search: keywords plus embeddings.

How does metadata filtering improve RAG accuracy?

The cheapest accuracy gain is to search less. If a question only concerns the 2026 pricing docs, searching the 2024 archive can only add noise. Attach metadata to every chunk at index time, then filter on it at query time so the search runs over the right subset.

Useful metadata fields:

• Source and section. The file path and heading the chunk came from, so you can scope a search to one product area.
• Date. When the content was written or last updated, so you can prefer fresh passages and exclude stale ones.
• Type. Whether the chunk is reference, a tutorial, an API description, or a changelog, so a "how do I" question can favour tutorials.
• Access. Which users or roles may see the chunk, so retrieval never returns content the asker is not allowed to read.

Store these alongside each chunk so a query can request, for example, only chunks where type = "tutorial" and date >= "2026-01-01". A pre-filterA filter applied before the similarity search, so the vector search only ever considers chunks that already match the metadata condition. narrows the candidate set first, which both speeds up the search and removes whole classes of wrong answers.

The metadata is only as good as the question's ability to use it. That is the next problem.

Sharpen the query

The user's words are rarely the best search query. They use pronouns, they ask two things at once, and they leave the topic implicit because they know what they mean. Three rewrites help.

Rewrite for retrieval

Turn a conversational question into a search-shaped one before you embed it. "What about the second one?" cannot be searched; "What is the refund window for the Pro plan?" can. Use the conversation so far to resolve pronouns and restore the dropped subject.

Split multi-part questions

"What does the API return and how do I authenticate?" is two searches wearing one question. Retrieving for the merged query finds passages that are mediocre on both topics and excellent on neither. Split it, retrieve for each part, and combine the results.

Expand sparse queries

A two-word query gives the embedding little to work with. Generating a few alternative phrasings and searching for all of them widens recall. This is one form of query expansionGenerating several reworded versions of a query and searching for all of them, then merging the results, so a poorly worded query still finds the right passages.; keep the variants close to the original intent so you widen recall without dragging in unrelated topics.

What does a reranker do in RAG?

Your first-pass search is built for speed over a large index, so it ranks approximately. A rerankerA second-pass model that reads the query and each candidate passage together and re-scores them, so the most relevant one moves to the top. reads the query and each candidate passage together and re-scores them, which is slower per item but far more accurate. So you run it on a shortlist, not the whole index.

The pattern is fetch wide, then rank narrow. Anthropic's tests retrieved the top 150 chunks, reranked them, and kept the top 20 for the model. Adding reranking on top of contextual hybrid search took the total top-20 failure reduction to 67 percent, from a 5.7 percent failure rate down to 1.9 percent. Reranking adds a runtime step, so it costs latency and money; rank fewer candidates if that budget is tight, more if accuracy matters more than speed.

How many chunks should I send to the model?

Sending more chunks is not safer. Two effects push back.

• Lost in the middle. Models attend best to the start and end of a long context and worst to the middle. Burying the key passage in the middle of forty chunks hides it. Send fewer, stronger chunks, and place the best one first or last.
• Distraction. Every irrelevant chunk is a chance for the model to anchor on the wrong fact. Precision in the final set matters as much as recall.

Tune the count to your task. Anthropic found top-20 best for its benchmark after reranking, but a focused question over clean docs can do better on five. Measure it; do not guess.

Measure retrieval on its own

You cannot improve what you do not measure, and a bad retriever feeding a good model looks exactly like a bad model. Score the retrieval stage separately.

• Recall asks whether the right chunks were fetched at all. This is the high-signal metric: if the answer is not in the retrieved set, the model cannot produce it.Common gotchaThe tempting shortcut is to grade only the final answer end to end and call that your accuracy number, but that mixes retrieval and prompting into one figure, so a hidden recall drop hides behind a forgiving model until it suddenly fails on a hard question.Score recall against a fixed question-to-chunk set on its own, separately from answer quality, so you can see which stage moved before and after every change.
• Precision asks how many fetched chunks were relevant. Low precision means noise that distracts the model.

Build a small evaluation set: real questions paired with the chunk that should answer each one. After every change to chunking, search, or filtering, re-run it and watch recall. Claude can help you build the set from your own documents.

A working order

When accuracy is low, change one thing at a time and re-measure. Doing everything at once tells you nothing about what worked.

1. Confirm it is a retrieval problem, not a prompt problem.
2. Fix chunking: respect structure, add overlap, make chunks self-contained.
3. Add metadata and pre-filter so each search runs over the right subset.
4. Add hybrid search so exact terms and meaning both count.
5. Rewrite and split queries before they hit the index.
6. Add a reranker over a wide shortlist.
7. Tune how many chunks reach the model, watching for lost-in-the-middle.

Each step is independent, and each is measurable against your evaluation set. Get chunking and retrieval right and the model has what it needs; the rest is grounding, which RAG Fundamentals covers. For the context window itself, see Context Engineering.