# RAG fundamentals: retrieval explained

Retrieval-augmented generation (RAG) is how you get a model to answer from your
data instead of only what it learned in training. You retrieve the relevant
pieces and hand them to the model as context. This guide is the pipeline from
first principles, and the failure modes that bite people in production.

## Why retrieve if I have a big context window?

A bigger context window does not remove the need for retrieval:

- **The window is not a knowledge base.** Even very large windows cannot hold a
  whole corpus, and stuffing everything in every call is wasteful and degrades
  quality.
- **Freshness.** Model weights are frozen at a training cutoff. An index can be
  updated continuously, so retrieval is how you serve new and private data.
- **Grounding.** Retrieved passages give the model something to cite, which is
  your main defence against confident wrong answers.
- **Cost.** Fetching the few relevant chunks is far cheaper per call than sending
  the whole corpus, even where it would fit.

## What are the stages of a RAG pipeline?

RAG is four stages, plus the answer.

**Chunking.** Split documents into passages. Fixed-size is simplest but blind to
structure. Recursive splitting (paragraph, then sentence, then word, until under
a size limit) is the common default. Semantic chunking splits at topic shifts and
is higher quality but more expensive. Add a small overlap, around ten to twenty
percent, so a fact on a boundary survives in at least one chunk.

**Embedding.** Turn each chunk into a vector
that captures its meaning. Pick a current, well-regarded embedding model; the
right one depends on your language, scale, and whether you are self-hosting.
Quality here sets the ceiling for everything downstream.You must embed the query with the exact same model and version that built the index. Mix two embedding models and the vectors live in different spaces, so cosine scores become noise and retrieval quietly returns garbage with no error. If you switch embedding models, re-embed the whole corpus before serving a single query.

**Storing.** Put the vectors in a [vector store](/guides/choose-a-vector-database). If you already run Postgres,
`pgvector` covers a large share of workloads. Dedicated stores trade setup for
speed, scale, and richer metadata filtering.

**Retrieving.** Embed the question, find the nearest chunks (cosine similarity is
standard), and take the top handful. Then ground: inject those chunks into the
prompt, instruct the model to answer only from them, and pass identifiers so it
can cite the source.Tell the model what to do when the chunks do not contain the answer, or it fills the gap from training and hands back a confident wrong answer that still looks grounded. Add an explicit instruction to reply "I don't know" when the retrieved context falls short.

## Why evaluate retrieval separately from the answer?

The single most useful habit in RAG is to evaluate the retrieval stage on its
own. Two ideas:

- **Recall vs precision.** Recall asks whether you fetched all the relevant
  chunks; precision asks how many of the fetched chunks were relevant.
  Recall is usually the higher-signal failure: if the answer is not in the
  retrieved context, the model cannot produce it.
- **Bad retrieval looks like a bad model.** A perfectly good model given the
  wrong context produces a wrong answer that is indistinguishable from a model
  failure, unless you [measure retrieval separately](/guides/evaluate-your-rag-system). Tools like RAGAS score this,
  with faithfulness and context recall being the highest-signal metrics to start
  from.

## Why does my RAG system give bad answers?

When RAG underperforms, it is usually one of these:

- **Stale index.** The index has drifted from the live source. Re-index
  incrementally as content changes.
- **Bad chunks.** Boundaries that split a fact or merge unrelated content hurt
  retrieval no matter how good the embeddings are.
- **Lost in the middle.** Models attend best to the start and end of a long
  context and worst to the middle. Retrieve fewer, better chunks, and put the
  strongest one first or last rather than burying it.
- **Missing exact terms.** Pure vector search can miss exact matches like IDs and
  error codes. [Hybrid search](/guides/hybrid-search-for-rag), combining keyword (BM25)
  with vector, catches both the literal and the semantic.
- **Weak ordering.** A reranker
  re-scores the top candidates by looking at the query and passage together, and
  reliably lifts answer quality for a small added latency.

Get chunking, retrieval, and evaluation right and RAG is dependable. The next
guide is about [a domain where the usual recipe quietly breaks: code](/guides/rag-for-coding-agents).