# Keep your knowledge base fresh

A retrieval index is a copy of your documents, and copies go stale. Keep it
accurate by re-indexing changed files, deleting chunks whose source is gone, and
running a sync on a schedule so the index never drifts far from the truth. This
guide shows the mechanisms and how to drive them from the Claude Code CLI.

## Why does a fresh index matter?

Retrieval answers from what is in the index, not from what is on disk. If a file
changes and the index does not, the model retrieves the old passage and answers
from it with full confidence. That is one of the most common ways a working
[RAG](/guides/rag-fundamentals) system quietly degrades.

Three kinds of drift cause wrong answers:

- **Updated content.** A doc changes but its old chunks remain. The model now
  has two versions of one fact and no way to tell which is current.
- **Deleted content.** A doc is removed from the source, yet its chunks survive
  in the index. The model cites a page that no longer exists.
- **Moved or renamed content.** The same passage gets indexed twice under two
  paths, so it crowds the top results and pushes out other relevant chunks.

Fresh-index work is the discipline of keeping the index a faithful mirror of the
source.

## How do I track which documents changed?

You cannot re-index sensibly until you can answer one question per document: has
this changed since the last run? The standard answer is a content hash.

Store, alongside each chunk in your vector store, metadata that ties it back to
its source:

- the source document id (a stable path or key)
- a content hash of the source document
- a timestamp of when it was indexed

On each sync you compute the current hash of every source document and compare
it to the stored hash. That comparison sorts every document into one of four
buckets.

This is why a hash beats a modified-time check. A file touched by a formatter
gets a new timestamp but the same hash, so you skip it and save the embedding
cost. A file restored from backup keeps an old timestamp but you still catch a
real content difference.

## Which sync mode should I use?

Pick by how much of the source you scan in one run. When you re-index, the
question is what to do with chunks that were there before, and most indexing
tools name three modes.

| Mode | What it touches | Deletes stale chunks? | Use when |
| --- | --- | --- | --- |
| Append | Adds new chunks only | No | One-off load, never re-run |
| Incremental | Adds new, replaces changed | Yes, per re-seen source | You scan a subset each run |
| Full | Adds, replaces, and prunes | Yes, anything not seen | You scan the whole source |

**Append** is the trap. It is the easy default and it is correct exactly once.
Run it twice on the same source and every chunk is now duplicated.

**Incremental** deletes the old chunks of any document it re-sees with a changed
hash, then writes the new ones. It cannot delete a document you did not scan,
because it has no way to know that document is gone. Use it when each run covers
only part of the corpus.

**Full** scans the entire source in one pass, so it knows the complete set of
live document ids. Anything in the index whose id is not in that set gets pruned.
This is the only mode that removes a document deleted from the source. Run it on
a schedule even if you re-index incrementally between runs.

## Deletions are the part people forget

Adding and updating are visible: someone edits a doc, the answer improves, the
loop feels complete. Deletion is silent. Nothing prompts you to remove the chunks
of a file that no longer exists, so they linger and keep getting retrieved.

Handle deletions one of two ways:

- **Periodic full sync.** Run a full re-index on a schedule. Because a full pass
  knows every live id, it prunes any chunk whose source is gone. Simple and
  reliable; costs a full scan.
- **Event-driven delete.** When your source system emits a delete event, delete
  that document's chunks by their source id immediately. Lower latency, but you
  depend on never missing an event, so pair it with a periodic full sync as a
  backstop.

A soft-delete variant keeps the chunks but flags them inactive and filters them
out at retrieval time. That lets you reverse a deletion and keep an audit trail,
at the cost of an index that grows until you prune the flagged rows.

## Drive the sync from the Claude Code CLI

You do not call an embedding model by hand for this. You give Claude the sync
script as a task and let it run, read the output, and react. The pattern is
"run, read, fix" inside the CLI.

```text
Run the knowledge-base sync in dry-run mode and read the output.
Tell me, in a short table: how many source docs are new, changed,
unchanged, and missing. If any source ids appear in the index but
not on disk, list them so I can confirm they should be pruned.
```

When the dry run looks right, have Claude run the real sync and confirm the index
matches the source:

```text
Run the full sync now. After it finishes, verify the index is
consistent: the number of distinct source ids in the vector store
should equal the number of source files on disk. Report any mismatch
with the offending ids.
```

If your vector store is reachable through an MCP server, Claude can query it
directly to check counts and spot orphaned chunks. See
[MCP fundamentals](/guides/mcp-fundamentals) for how that connection works and
[install your first MCP server](/guides/install-your-first-mcp-server) to set one
up.

## How do I re-index automatically when docs change?

You can make freshness a side effect of editing, so the index never waits for a
human to remember. A hook
in your project settings runs a command after Claude writes or edits a file.
Point it at your sync script.

Add this to your project settings file (for example `.claude/settings.json`):

```json
{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Write|Edit",
        "hooks": [
          {
            "type": "command",
            "command": "node scripts/sync-index.js --incremental --changed-only"
          }
        ]
      }
    ]
  }
}
```

The `matcher` field selects which tools fire the hook; `Write|Edit` runs the
command after any file write or edit.That matcher fires on every write, including code and config files that are not part of your corpus, so an unscoped hook re-embeds on edits that changed nothing in the index. Have the script exit early when the changed path is outside your docs directory. Keep this hook incremental and scoped to
changed files so it stays cheap on every save. A `PostToolUse` hook runs after
the tool finishes and cannot undo it, so use it to react to a change, not to
gate one.

For the full-prune pass, do not put it on every edit. Run it on a timer.

A nightly cron entry that calls the same script in full mode keeps deletions and
any missed edits in check:

```bash
# Re-index the whole corpus every night at 02:30
30 2 * * * cd /srv/app && node scripts/sync-index.js --full >> /var/log/index-sync.log 2>&1
```

## How do I reduce re-embedding cost?

Re-embedding is the expensive step, so structure things to re-embed as little as
possible:

- **[Chunk by stable boundaries](/guides/chunk-your-documents).** Split on
  headings and sections, not a fixed character count. When one section of
  `docs/checkout-flow.md` changes, only its chunks change hash; the rest skip
  re-embedding.
- **Hash per chunk, not only per document.** A document-level hash re-embeds the
  whole file on any edit. A per-chunk hash re-embeds only the chunks that moved.
- **Keep the source id stable across renames.** If you key chunks on a file path,
  a rename looks like a delete plus an add and re-embeds everything. Key on a
  stable id where you can, so a rename is a metadata update.

## Check the index matches the source

Trust the sync only after you measure it. Two checks catch most drift:

- **Count parity.** Distinct source ids in the index equal source files on disk.
  A surplus means stale chunks survived a deletion; a deficit means a sync
  failed partway.A bare count match can lie: if one document got pruned by mistake and an unrelated stale doc survived, the two cancel and the totals still tie while both rows are wrong. Compare the actual set of source ids, not just the counts, so you catch an id present in one place and absent from the other.
- **Spot-retrieval.** Edit one doc, sync, then ask a question whose answer lives
  only in the new version. If the old answer comes back, your update path is
  broken, not your model.

Build the first check into the sync output and have Claude read it after every
run. Drift is easy to introduce and hard to notice, so make the consistency
number visible rather than assumed.

## What good freshness looks like

A healthy knowledge base has three loops running:

- a fast incremental sync on edit, so new and changed content is searchable in
  seconds
- a scheduled full sync, so deletions and missed edits cannot accumulate
- a consistency check after each run, so drift surfaces as a number, not as a
  bad answer weeks later

Get those three in place and your retrieval stays as current as your docs. For
the stage that runs against this index, read
[RAG fundamentals](/guides/rag-fundamentals) on retrieving and grounding from
what you have kept fresh.