How to Test a RAG Application: Verifying Retrieval-Augmented Generation

RAG (Retrieval-Augmented Generation) is now the most common architecture for enterprises adopting LLMs — first pull relevant material from your own documents and knowledge base, then have the model answer based on that material. But RAG introduces a problem traditional QA isn't used to: when an answer is wrong, you first have to figure out whether it "retrieved the wrong material" or "the material was right but the model got it wrong." This article covers how to layer the quality verification of a RAG application.

Why you can't just test "is the final answer right"

A RAG answer is two steps chained together: retrieval finds relevant document chunks, and generation produces an answer based on them. Look only at the final answer and you hit a problem — the answer's wrong, but you don't know which step to fix. To verify RAG effectively, you have to test retrieval and generation separately, then test how they join.

Three layers of RAG quality verification

Layer	What it verifies	Typical failure
Retrieval	Are the retrieved documents actually relevant to the question?	Missing what should've been retrieved, or pulling irrelevant chunks
Generation	Did the model answer faithfully based on what was retrieved?	The material was there, but the answer is wrong or invented
End-to-end	From user question to final answer, is it right overall?	Each layer's fine, but combined it misses the point

1. Retrieval layer: relevance

First verify that, given a question, the chunks the system retrieves are actually relevant. This layer can be measured in a fairly traditional way — prepare a set of questions, label which documents each one "should retrieve," and check the system's actual hit rate. The cost of retrieving wrong is high: if the correct material isn't even retrieved, no model, however strong, can answer correctly.

2. Generation layer: faithfulness

Even with the right material retrieved, the model can still "fail to answer faithfully based on it" — adding details not in the source, or incorrectly merging content from two documents. The core question here is: can every claim in this answer be grounded in the retrieved material? This is the behavior eval we cover in "How to QA an LLM Feature" — not comparing text, but checking whether behavior meets expectations.

3. End-to-end: overall quality

Finally, verify the full flow from the user's perspective: did the answer actually address the question, is the tone right, did it cite sources? This layer is closest to the real user experience, but also needs the most human judgment, because a "good answer" often has several valid forms.

Three scenarios where RAG is especially prone to problems

• Old answers silently going stale after a knowledge base update: documents change, but nobody goes back to verify "do the questions we tested before still answer correctly?" This is the same failure mode as test case drift — the source of drift this time is the knowledge base.
• Behavior when nothing is retrieved: when the knowledge base genuinely has no answer, the model should honestly say "I couldn't find it" rather than fabricate one. This must be in the test set.
• Boundary and adversarial scenarios: when someone asks something outside the knowledge base's scope, or deliberately probes, does the system hold? This connects to red-teaming.

The knowledge base changes, so RAG testing has to change with it

The most overlooked thing about a RAG application: its quality depends not just on the model, but on the constantly-updating knowledge base behind it. Documents get added, edited, deleted — and the matching verification cases may all need re-examining. This is exactly where OpenTestX helps — treating a knowledge base or requirement change as an active signal, flagging "this change might affect these existing RAG verification cases" for a human to review, instead of finding out only when a user asks a question that gets a stale answer. Eval Studio also supports authoring behavior checks for AI features like this, comparing variant performance, and keeping reviewable evidence.

FAQ

Which layer causes problems more often — retrieval or generation?

In practice, retrieval problems are often more hidden and more critical — because if the correct material isn't even retrieved, no amount of model tuning downstream can recover it. Confirm the retrieval layer's relevance is high enough first, then tune the generation layer.

Can RAG faithfulness be evaluated automatically?

You can do preliminary automated checks (e.g., whether claims in the answer appear in the retrieved sources), but the fuzzy judgment of "does this count as faithful" is still worth keeping under human review — same reason regular AI-generated content needs human review.

How often should you re-run RAG tests when the knowledge base updates?

The trigger is whether the knowledge base changed, not a fixed schedule. Whenever a document that affects existing test questions is modified, it's worth re-verifying those questions — same logic as re-checking test cases after a requirement change.

---

The key to testing a RAG application is separating "retrieving material" from "producing the answer." Only by pinpointing which layer failed can you fix the right place; and the fact that the knowledge base keeps changing means RAG verification isn't a one-time task, but one you have to keep up with.