AIF-C01 practice question: Evaluating foundation models (2 of 3)
Applications of Foundation Models A free question from our original AIF-C01 practice exam. Try it, then open the answer to see why each option is right or wrong.
A support team has 5,000 chatbot answers from last month.
It wants each answer scored for correctness and completeness, applied consistently, without reviewers reading every answer by hand. The team plans to run the evaluation on Amazon Bedrock.
Which approach fits BEST?
Choose one answer.
AA. Create a Bedrock model evaluation job with a judge model and the built-in Correctness and Completeness metrics.
BB. Compute a BLEU score for each answer against a stored reference answer, then sort all the answers by that score.
CC. Ask reviewers to read a random 50 answers and apply their average score to the full set of 5,000 answers.
DD. Count the thumbs-up and thumbs-down clicks customers gave each answer, and rank the answers by total.
B. Wrong: BLEU measures word overlap with a reference. It does not check whether the facts are correct or whether the answer covers everything the customer needed.
C. Wrong: A sample gives only an estimate for the whole set, and the team wants a score for every answer. Reviewers would still be reading many answers by hand.
D. Wrong: Clicks reflect how users felt, not whether the answer is correct or complete. Many answers will also have no votes at all.
Explanation
Judge-model evaluation on Amazon Bedrock uses a model to score responses against chosen metrics, which makes large-scale quality checks practical. Overlap scores and customer clicks measure different things. This question tests choosing an evaluation method that matches both the criteria and the volume.
Want more on Evaluating foundation models? The 5 full-length practice sets have 21 more Evaluating foundation models questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →