Evaluating RAG and retrieval quality: AIP-C01 study notes and practice questions
What to know about Evaluating RAG and retrieval quality for the AIP-C01 exam, from the Testing, Validation, and Troubleshooting domain (11% of the exam guide), then 1 free question to test it. Part of the AIP-C01 study guide.
Split RAG evaluation in two. If retrieval fails, no change to the generator will fix the answer, so measure retrieval first and then measure the answer.
Retrieve-only evaluation
Retrieve-only evaluation scores the retrieved chunks without generating an answer. Context relevance asks whether each chunk relates to the question. Context coverage asks whether the chunks together contain the information needed. Use it to tune chunking, embeddings, filters and reranking, because it isolates retrieval from generation.
Retrieve-and-generate evaluation
Retrieve-and-generate evaluation scores the final answer. Correctness compares the answer with a reference. Completeness checks that the answer covers the question. Faithfulness checks that the answer uses only the retrieved text, which catches hallucinations. Citation precision and coverage check that the cited sources support the claims. Use it for end-to-end release checks.
Golden set and latency
Build a golden set of questions, reference answers and the source documents each answer should use. Include easy, hard and ambiguous questions, and out-of-scope questions that the system should decline. Measure retrieval latency separately from generation latency, because the slow step may be the search.
Compare chunking configurations
To compare chunking strategies, ingest the same corpus under each configuration into separate stores or index versions. Run the same golden set against each, and compare context coverage, faithfulness and retrieval latency. Changing chunking usually means re-ingesting, so test on a representative subset before you rebuild everything.
How to choose
Scenario wording
Choose
The right document is never retrieved
Retrieve-only evaluation of context relevance and coverage
The answer invents facts not in the sources
Retrieve-and-generate evaluation with faithfulness
The answer is right but cites the wrong sources
Citation precision check
Compare chunk sizes on the same questions
Golden set run once per configuration
Users say search is slow
Measure retrieval latency on its own
The system should refuse questions the corpus cannot answer
Out-of-scope questions in the golden set
Common traps
Judging only the final answer, which hides a retrieval failure underneath.
A golden set of easy questions that every configuration passes.
Judging faithfulness without giving the judge the retrieved context.
Comparing chunk settings on different questions, which proves nothing.
Treating the generator’s own confidence as a faithfulness measure.
A pharmacy benefits company runs a knowledge base that answers members' questions about formulary rules. The team has changed the chunking strategy and the embedding model, and it wants to know whether the retriever now returns the right passages before the answers are checked.
The team has 300 questions, each with a reference answer that lists the facts a good response must contain. The evaluation must look only at retrieval, so it should not depend on the generation model's wording. It must measure whether the retrieved passages are relevant to each question and whether they cover the facts in the reference answers.
Which evaluation approach meets these requirements?
Choose one answer.
AA. Run a RetrieveAndGenerate evaluation job that scores final answers for correctness and faithfulness, and compare scores before and after the change.
BB. Use an LLM as a judge to rate each final chatbot answer on a custom rubric for helpfulness, and track the average rating for each chunking option.
CC. Run a retrieve-only RAG evaluation job with context relevance and context coverage metrics on the 300 questions, with reference answers as ground truth.
DD. Ask pharmacists to read the final answers to all 300 questions and mark any that are wrong, then count the errors for each chunking option.
Show the answer and explanation
Answer: C
A. Wrong: Correctness and faithfulness depend on the generation model's answers, so the score mixes retrieval and generation quality. The team needs a score for the retrieved passages alone.
B. Wrong: A final answer rating blends retrieval with generation, and the rubric is custom work the team does not need. A weak answer could come from either part of the system.
D. Wrong: Reading final answers mixes retrieval errors with generation errors, and it cannot say which passage was missing. The manual effort also takes far longer than a job that runs on its own.
Explanation
Retrieve-only RAG evaluation scores the passages returned for each question, so the team can see whether the retriever finds the relevant text and covers the reference facts. The generation model's wording is excluded from the score. Scoring final answers, with a custom judge or with human reviewers, blends the two stages together.
Want more on Evaluating RAG and retrieval quality? The 5 full-length practice sets have 5 more Evaluating RAG and retrieval quality questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →