Room 42, home

RAG fundamentals: retrieval explained

Retrieval-augmented generation from first principles: chunk, embed, retrieve and ground the answer, then evaluate what you built.

RAGadvanced4 min readUpdated 12 Jun 2026

In short

  1. 01RAG gets a model to answer from your own data by fetching the relevant pieces and handing them over as context.
  2. 02The pipeline is four stages: chunk your documents, embed them, retrieve the closest chunks, then ground the answer in them.
  3. 03Evaluate retrieval on its own, since bad context produces a wrong answer that looks exactly like a model failure.
  4. 04Watch for stale indexes, bad chunks, lost-in-the-middle ordering, and missing exact terms that pure vector search drops.
Contents · 4 sections
  1. 1Why retrieve if I have a big context window?
  2. 2What are the stages of a RAG pipeline?
  3. 3Why evaluate retrieval separately from the answer?
  4. 4Why does my RAG system give bad answers?

Retrieval-augmented generation (RAGA pattern where you fetch relevant text from your own data and feed it to the model as context, so it answers from that instead of only its training.) is how you get a model to answer from your data instead of only what it learned in training. You retrieve the relevant pieces and hand them to the model as context. This guide is the pipeline from first principles, and the failure modes that bite people in production.

Why retrieve if I have a big context window?

A bigger context window does not remove the need for retrieval:

  • The window is not a knowledge base. Even very large windows cannot hold a whole corpusThe entire collection of documents you want the system to be able to draw answers from., and stuffing everything in every call is wasteful and degrades quality.
  • Freshness. Model weights are frozen at a training cutoff. An index can be updated continuously, so retrieval is how you serve new and private data.
  • Grounding. Retrieved passages give the model something to cite, which is your main defence against confident wrong answers.
  • Cost. Fetching the few relevant chunks is far cheaper per call than sending the whole corpus, even where it would fit.

What are the stages of a RAG pipeline?

RAG is four stages, plus the answer.

Chunking. Split documents into passages. Fixed-size is simplest but blind to structure. Recursive splitting (paragraph, then sentence, then word, until under a size limit) is the common default. Semantic chunking splits at topic shifts and is higher quality but more expensive. Add a small overlap, around ten to twenty percent, so a fact on a boundary survives in at least one chunk.

Embedding. Turn each chunk into a vectorA list of numbers that captures the meaning of a piece of text, so that two passages about the same thing land close together in that number space. that captures its meaning. Pick a current, well-regarded embedding model; the right one depends on your language, scale, and whether you are self-hosting. Quality here sets the ceiling for everything downstream.Common gotchaYou must embed the query with the exact same model and version that built the index. Mix two embedding models and the vectors live in different spaces, so cosine scores become noise and retrieval quietly returns garbage with no error.If you switch embedding models, re-embed the whole corpus before serving a single query.

Storing. Put the vectors in a vector store. If you already run Postgres, pgvector covers a large share of workloads. Dedicated stores trade setup for speed, scale, and richer metadata filtering.

Retrieving. Embed the question, find the nearest chunks (cosine similarityA way to measure how similar two vectors are by the angle between them, so passages pointing in the same direction count as close in meaning. is standard), and take the top handful. Then ground: inject those chunks into the prompt, instruct the model to answer only from them, and pass identifiers so it can cite the source.Common gotchaTell the model what to do when the chunks do not contain the answer, or it fills the gap from training and hands back a confident wrong answer that still looks grounded.Add an explicit instruction to reply "I don't know" when the retrieved context falls short.

Why evaluate retrieval separately from the answer?

The single most useful habit in RAG is to evaluate the retrieval stage on its own. Two ideas:

  • Recall vs precision. Recall asks whether you fetched all the relevant chunks; precision asks how many of the fetched chunks were relevant. Recall is usually the higher-signal failure: if the answer is not in the retrieved context, the model cannot produce it.
  • Bad retrieval looks like a bad model. A perfectly good model given the wrong context produces a wrong answer that is indistinguishable from a model failure, unless you measure retrieval separately. Tools like RAGAS score this, with faithfulness and context recall being the highest-signal metrics to start from.

Why does my RAG system give bad answers?

When RAG underperforms, it is usually one of these:

  • Stale index. The index has drifted from the live source. Re-index incrementally as content changes.
  • Bad chunks. Boundaries that split a fact or merge unrelated content hurt retrieval no matter how good the embeddings are.
  • Lost in the middle. Models attend best to the start and end of a long context and worst to the middle. Retrieve fewer, better chunks, and put the strongest one first or last rather than burying it.
  • Missing exact terms. Pure vector search can miss exact matches like IDs and error codes. Hybrid search, combining keyword (BM25A classic keyword-ranking algorithm that scores documents by exact word matches and how rare those words are. Strong on exact terms, blind to meaning.) with vector, catches both the literal and the semantic.
  • Weak ordering. A rerankerA second-pass model that re-scores the top retrieved chunks by reading the query and each chunk together, so the most relevant one ends up first. re-scores the top candidates by looking at the query and passage together, and reliably lifts answer quality for a small added latency.

Get chunking, retrieval, and evaluation right and RAG is dependable. The next guide is about a domain where the usual recipe quietly breaks: code.