How to evaluate a RAG system before you ship it

A RAG system can fail in two places, and a single "does it look right?" review hides which one. Retrieval can miss the passage that holds the answer. Or retrieval can succeed and the model can ignore the passage, add claims it does not support, or answer a different question. Evaluate the two stages separately. If you are still deciding whether RAG is the right approach, read RAG vs fine-tuning first.

Build the evaluation set first

Everything else depends on a set of test cases. Build it from real user questions, not from questions the team invents while reading the documents.

• Start with about twenty real questions. That is a working suggestion, not a statistical guarantee, so add more as you find failures.
• For each question, record the answer you would accept and the document passage that supports it.
• Include questions the documents cannot answer. The correct behavior is to say so, and models sometimes answer anyway.
• Include permission cases. A user asks about a document they may not see, and the system must not reveal it.

What a few rows look like

These rows are made up to show the shape. Each row records the question, the answer you would accept, the passage that supports it, and the case type.

• Question: "How long do customers have to request a refund on an annual plan?" Accepted answer: the number of days stated in the policy. Supporting passage: the refund policy section. Type: answerable.
• Question: "Do you offer a discount for nonprofits?" Accepted answer: a statement that the documents do not say, with a pointer to sales. Supporting passage: none. Type: unanswerable.
• Question from a support agent: "What is the CEO's salary?" Accepted answer: a refusal. Supporting passage: an HR document that the agent's role cannot access. Type: permission.

Keep the set under version control next to the code. Treat each new failure found in production as a new test case.

Test retrieval on its own

Before any model writes an answer, check what the retriever returns for each test question. Check whether the supporting passage appeared in the results and how high it ranked. A simple measure is the share of test questions where the supporting passage appears in the top few results.

If that number is low, a prompt or model change cannot recover a passage the retriever never returned. Change how documents are split into chunks, change the embedding model, or try keyword search alongside vector search, then measure again.

Test generation separately

Given the retrieved passages, does the answer stay within them? The Ragas paper names three dimensions worth separating: whether retrieval returns relevant and focused passages, whether the model uses those passages faithfully, and the quality of the generated answer itself (Es et al., 2023). The same paper describes metrics that need no human-written reference answers, which its authors say can shorten evaluation cycles. Use them for fast feedback, and keep human-checked answers for the launch decision.

Check the judge

Automated scoring often uses a second language model as a judge. That is useful, and it has known weaknesses. Research on LLM judges documents position bias, verbosity bias, and self-enhancement bias, along with limited reasoning ability (Zheng et al., 2023). Have a person read a sample of judged answers each time you change the judge prompt or model, and compare the two sets of scores. Two cheap checks target the biases above: swap the order of the two answers when the judge compares them and see whether the verdict changes, and compare its scores on short and long answers to the same question.

Mind the number and order of passages

Research on long inputs found that model performance can drop when the relevant information sits in the middle of a long context, and is often highest at the start or end (Liu et al., 2023). Test different numbers of retrieved passages and different orderings against your own set rather than assuming the largest window wins. One direct test: for the same questions, place the supporting passage first, in the middle, and last among the retrieved passages, and compare accuracy by position.

Set the launch threshold in advance

Agree on the numbers before you look at results. For example: retrieval finds the supporting passage for a stated share of questions, answers contain no unsupported claims in a human-reviewed sample, the system declines every unanswerable test question, and no permission case leaks. The right values depend on what an error costs in your setting. Write them down before the scores arrive.

Run it on every change

Changes to chunking, embeddings, prompts, the model, or the document set can all move the scores. Run the evaluation set automatically on each one and keep the history. A drop after a change shows you what caused it. Without the history, a user is often the first to notice.

Keep learning after launch

Log the questions where retrieval returned nothing useful, and the answers users flag. Review them weekly and add the real failures to the set. That log is a steady source of new test cases.

Where to go from here

If you are building a RAG system and want it measured this way, see our RAG development services. For the monitoring and cost work after launch, see MLOps and production deployment. For the wider picture, start with what generative AI development involves.