Retrieval-augmented generation, or RAG, answers a question in two steps. First a search step finds the passages in your documents most likely to hold the answer. Then the model writes its reply from those passages, ideally citing them. The model learns nothing permanently. The answer is only as good as what retrieval found, so most of the work is in search, not the model.
What happens when a RAG system answers a question?
Six steps; the first two happen ahead of time.
- Split. Documents are cut into chunks, each tagged with its source, date, owner and who may see it.
- Index. Each chunk is stored twice: as an embedding, a list of numbers that captures meaning, and in a keyword index such as BM25, which matches exact words.
- Retrieve. The question is embedded and searched against both indexes, and the best matches are merged.
- Rerank. A second model rescores the candidates against the question and keeps the best few.
- Assemble. The chosen chunks go into the prompt with instructions: answer only from these sources, cite them, say so if the answer is not there.
- Generate. The model writes the answer from that context, token by token, as it always does.
Do you need RAG at all?
Often not. RAG pays off when the material is too large to hand the model whole, or changes too often to curate by hand.
- Small, stable material: put it in the prompt. Anthropic's guidance is that a knowledge base under about 200,000 tokens, roughly 500 pages, can simply be included, and prompt caching keeps the repeat cost down. A Claude Project does this without code.
- Live systems: give the assistant a search tool instead of an index. A connector that queries the CRM returns current records.
- Large, mixed document sets: this is where a retrieval pipeline pays off.
Why does keyword search still matter?
Embeddings find passages that mean the same thing in different words. They are weak on exact strings: error codes, part numbers, clause numbers, a person's name. Keyword search is the opposite. Use both.
Anthropic's Contextual Retrieval research, published in September 2024, measured this. Adding a short note of document context to each chunk before indexing cut top-20 retrieval failures by 35%. Adding keyword search on top took it to 49%, and adding a reranker to 67%. Your numbers will differ.
What makes an answer grounded?
- Every claim traces to a passage. Show citations to the source and section, so the reader can check in one click. Claude's API can return citations that point to the exact passage it used.
- "Not in the sources" is an allowed answer. Instruct the model to say so, and test that it does.
- Permissions are applied at retrieval. Filter chunks by who is asking before they reach the prompt. A model cannot un-see a salary file it was shown.
- Dates travel with the text. Put the document date in the chunk so the model can prefer the current policy over last year's.
Where does RAG go wrong?
- Chunks cut in the wrong place: a table split from its header, a clause split from its exceptions.
- Stale and duplicate documents: three versions of the handbook, and retrieval picks the oldest.
- Too little retrieved: the answer was in chunk 12 and only 5 were kept.
- The model fills gaps from general knowledge when retrieval came back thin.
- Instructions hidden in documents: a retrieved page can contain text aimed at the model. Treat retrieved content as data, and keep risky actions out of reach of a system that reads it.
Find these by measuring, not by reading demo answers. RAG evaluation for internal knowledge bases covers the test set.
Common questions
What is the difference between RAG and fine-tuning?
RAG gives the model relevant documents at the moment it answers, so updating knowledge means updating the documents. Fine-tuning changes the model's weights with training examples, which suits style and format better than facts. For company knowledge that changes, RAG or a live search tool is almost always the right choice.
Does RAG stop AI hallucinations?
It reduces them but does not eliminate them. The model can still misread a passage or fill a gap when retrieval returns little. Require citations, allow the answer that the sources do not say, and measure faithfulness on real questions.
Do I need a vector database for RAG?
Not always. Small collections can go straight into the prompt, and many search tools already combine keyword and semantic search. A dedicated vector store earns its place when you have a large, changing document set and need control over chunking, metadata and permission filters.