Why RAG answers fail: chunking and retrieval

It is tempting to blame the model when a RAG system gives a wrong answer. Some of the documented failure points sit earlier in the pipeline. This post concentrates on those. Barnett and colleagues studied RAG systems in three case studies, in research, education, and biomedical settings, and reported seven failure points. Their two main takeaways were that a RAG system can be validated only during operation, and that its robustness evolves over time instead of being designed in at the start (Barnett et al., 2024).

The seven failure points

1. Missing content. The question cannot be answered from the documents, and the system answers anyway instead of saying it does not know.
2. Missed top-ranked documents. The answer is in a document, but the document did not rank high enough to be returned.
3. Not in context. Documents with the answer were retrieved but did not make it into the context the model saw, because of how the results were consolidated.
4. Not extracted. The answer is in the context, but the model fails to pull it out, typically when the context has too much noise or contradicting information.
5. Wrong format. The question asked for a table or a list and the model ignored the instruction.
6. Incorrect specificity. The answer is too general or too specific for what the user needs.
7. Incomplete. The answer is correct but leaves out information that was in the context. The paper suggests asking such questions separately.

In our reading, the first three sit at or before retrieval and the rest at generation, so check them in that order. A fault upstream hides the ones behind it, and a better prompt cannot recover a passage the retriever never returned.

Chunking

Before search can work, documents are split into chunks. Anthropic says chunks are usually no more than a few hundred tokens. Chunking can destroy context. In Anthropic's example, a chunk reads "The company's revenue grew by 3% over the previous quarter." On its own it does not say which company or which period, so a question about one company's second-quarter growth may never retrieve it (Anthropic, 2024).

Anthropic's fix, which it calls contextual retrieval, prepends a short explanation of where the chunk sits in its document to each chunk before it is embedded and indexed. Anthropic reports that contextual embeddings combined with a contextual BM25 index reduced the top-20-chunk retrieval failure rate by 49% (from 5.7% to 2.9%), and that adding reranking brought the reduction to 67% (to 1.9%). Those are Anthropic's own results on its own tests. Try the technique on your documents before you count on it.

Some practical habits follow. Split on the document's structure, such as headings and sections, instead of at a fixed character count. Keep the document title and section heading with every chunk. Try two or three chunk sizes on your evaluation set. Chunks that are too small lose the context that gives them meaning. Chunks that are too large mix topics and take up more of the prompt.

Keyword search and vector search

Vector search finds passages with similar meaning, and keyword search finds passages with the same words. Each wins in some settings. The dense passage retrieval paper reported that a dense retriever beat a strong BM25 keyword system by roughly 9 to 19 percentage points of top-20 retrieval accuracy on open-domain question answering datasets (Karpukhin et al., 2020). The BEIR benchmark later tested ten retrieval systems on 18 datasets and found BM25 a robust baseline, with dense models often underperforming when tested outside the data they were trained on (Thakur et al., 2021).

Both findings hold for the settings tested, and your documents may behave differently. Many teams combine the two kinds of search, and Anthropic's contextual retrieval does the same, pairing contextual embeddings with a contextual BM25 index. Keyword search helps most on exact strings such as product codes, names, and error messages.

Reranking and how much to send

A common pattern retrieves a wide set of candidates and then reranks them to choose the best few. Anthropic's 67% figure above includes reranking, and BEIR found that re-ranking and late-interaction models achieved the best zero-shot results on average, at high computational cost. Sending a few well-chosen passages costs less than sending many, and position matters too. In tests on 2023-era models, Liu and colleagues found that models often use information less reliably when it sits in the middle of a long input (Liu et al., 2023), so order the passages deliberately and test the order.

When you may not need retrieval

In 2024, Anthropic noted that if your knowledge base is smaller than 200,000 tokens, about 500 pages, you can include all of it in the prompt with no retrieval, and prompt caching makes that faster and cheaper. Check the size of your corpus, your model's context window, and the cost of a long prompt before you build a retrieval pipeline. Because of the position effect above, still test whether answers hold up when the prompt is that long.

How to find your failure

Start from the evaluation set described in our post on evaluating a RAG system. For each wrong answer, check in order:

1. Is the answer in the documents at all?
2. Did retrieval return the supporting passage, and how high did it rank?
3. Did that passage reach the final prompt?
4. Did the model use it correctly?

Tally the failures by point, fix the largest group, and change one thing at a time so you can tell what helped. Since the Barnett paper found that validation is possible only during operation, keep logging real questions and add each new failure to the test set. Testing before launch catches the failures you can anticipate, and logging in operation catches the rest. Costs move with these choices too, and our post on running costs shows how. Our RAG development page describes the service and says when it is the wrong choice.