ExamPark
Course menuAIP-C01

Exams / AIP-C01 / Evaluating RAG and retrieval quality / Question 1

AIP-C01 practice question: Evaluating RAG and retrieval quality

Testing, Validation, and Troubleshooting A free question from our original AIP-C01 practice exam. Try it, then open the answer to see why each option is right or wrong.

Last verified against current AWS documentation ·How we write our questions

A pharmacy benefits company runs a knowledge base that answers members' questions about formulary rules. The team has changed the chunking strategy and the embedding model, and it wants to know whether the retriever now returns the right passages before the answers are checked.

The team has 300 questions, each with a reference answer that lists the facts a good response must contain. The evaluation must look only at retrieval, so it should not depend on the generation model's wording. It must measure whether the retrieved passages are relevant to each question and whether they cover the facts in the reference answers.

Which evaluation approach meets these requirements?

Choose one answer.

  1. A. Run a RetrieveAndGenerate evaluation job that scores final answers for correctness and faithfulness, and compare scores before and after the change.
  2. B. Use an LLM as a judge to rate each final chatbot answer on a custom rubric for helpfulness, and track the average rating for each chunking option.
  3. C. Run a retrieve-only RAG evaluation job with context relevance and context coverage metrics on the 300 questions, with reference answers as ground truth.
  4. D. Ask pharmacists to read the final answers to all 300 questions and mark any that are wrong, then count the errors for each chunking option.
Show the answer and explanation

Answer: C

  • A. Wrong: Correctness and faithfulness depend on the generation model's answers, so the score mixes retrieval and generation quality. The team needs a score for the retrieved passages alone.
  • B. Wrong: A final answer rating blends retrieval with generation, and the rubric is custom work the team does not need. A weak answer could come from either part of the system.
  • C. Right: Retrieve-only evaluation scores the passages without invoking the generator, so the wording of answers does not matter. Context relevance and context coverage measure the two things the team asked about. Learn more: Evaluate the performance of RAG sources using Amazon Bedrock evaluations (opens in a new tab)
  • D. Wrong: Reading final answers mixes retrieval errors with generation errors, and it cannot say which passage was missing. The manual effort also takes far longer than a job that runs on its own.

Explanation

Retrieve-only RAG evaluation scores the passages returned for each question, so the team can see whether the retriever finds the relevant text and covers the reference facts. The generation model's wording is excluded from the score. Scoring final answers, with a custom judge or with human reviewers, blends the two stages together.

Study

Keep practicing

Answer questions with a score at the end, or practice just this topic with the explanation after each one. No sign-up needed.

Take the free AIP-C01 practice examPractice Evaluating RAG and retrieval qualityAll Evaluating RAG and retrieval quality questions

Want more on Evaluating RAG and retrieval quality? The 5 full-length practice sets have 5 more Evaluating RAG and retrieval quality questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets