To evaluate RAG on an internal knowledge base, collect real questions with the source that answers each one, then score two things separately. Did retrieval find the right passage, and did the answer stay faithful to it? Include questions the system should decline. Rerun the same set after every change to prompts, models, chunking or permissions, and treat a drop as a failed release.
What should a RAG evaluation measure?
Two stages, scored apart, because they fail for different reasons. If the right passage never reached the model, no prompt change will help.
| Metric | Question it answers | How to score it |
|---|---|---|
| Recall at k | Was the right source in the top k results? | Automatic, against the expected source IDs |
| Context precision | How much of what was retrieved was relevant? | Model grader or labels |
| Faithfulness | Is every claim in the answer supported by the retrieved text? | Model grader, checked against people |
| Completeness | Did the answer include the facts it had to? | Checklist of required facts per question |
| Citation accuracy | Do the citations point at the passage that supports the claim? | Automatic where citations carry IDs |
| Correct refusal | Did it decline when the sources do not answer, or the asker may not see them? | Automatic, on the unanswerable items |
Open-source libraries such as Ragas implement several of these. Names differ between tools, so define each metric in your own report.
How do you build a golden set?
From real questions: the help desk, the chat channel where people ask "does anyone know", and the assistant's own logs.
- Start with about 50 questions and grow toward a few hundred as failures teach you what to add.
- Cover the hard cases on purpose: exact identifiers, questions whose answer changed recently, ambiguous wording, answers spread across two documents, and questions with no answer in the sources.
- Add permission cases: a question only managers may have answered, asked as someone who is not a manager.
- Give every item an owner who can say what the right answer is, and a review date so the set does not go stale with the documents.
id: pto-carryover-2026
question: How many unused PTO days can I carry into next year?
expected_sources: [handbook-2026#time-off]
must_include: ["5 days", "must be used by March 31"]
must_not_include: ["10 days"] # the 2025 policy
asked_as: employee
owner: people-ops
review_by: 2027-01-15Can a language model grade the answers?
Yes, for faithfulness and relevance, if you check it. A model grader reads the question, the passages and the answer and scores against a written rubric, cheaply enough to run on every change.
- Calibrate it. Have a person score a sample of 30 to 50 items and compare. Where the grader and the person disagree, fix the rubric.
- Make it show its work. Ask for the unsupported sentence and the passage it checked, not just a score.
- Prefer exact checks where they exist. Whether a source ID is in the top five does not need a model to judge.
- Keep the grader fixed between runs you compare. A new grader model changes the scores by itself.
How do you turn evaluation into a release check?
Run the set before any change to the prompt, the model, chunking, the index, the retriever or source permissions. Compare against the last accepted run.
- Block the release if any critical item regresses, such as a permission case or a safety answer.
- Block it if a headline metric drops past a threshold you set in advance.
- Report what changed, the metric deltas, the items that newly pass and newly fail, and who owns each failure.
What does each failure usually mean?
| Symptom | Likely cause | Owner |
|---|---|---|
| Right source never retrieved | Chunking, missing keyword search, bad metadata | Retrieval |
| Old policy cited | Stale or duplicate documents in the index | Content owner |
| Source retrieved, answer wrong | Prompt, too much noise in context, model | Application |
| Answers when it should decline | No "not in the sources" instruction, or it is not tested | Application |
| Restricted content in an answer | Permissions filtered after retrieval, or not at all | Security |
Why keyword search and reranking move these numbers is in how RAG works.
Common questions
How many questions do you need to evaluate a RAG system?
Start with about 50 real questions that cover common requests, exact identifiers, recent changes and questions with no answer in the sources. Grow toward a few hundred as production failures show what is missing.
What is faithfulness in RAG evaluation?
Faithfulness measures whether every claim in an answer is supported by the passages the system retrieved. An answer can be correct in general and still unfaithful, if it adds facts the sources do not contain. It is the main measure of whether the system is grounded.
How often should you run a RAG evaluation?
Before every change to the prompt, model, chunking, index, retriever or source permissions, and on a schedule, because the documents change too. Treat a drop on critical questions as a failed release.