ExamPark
Course menuMLA-C02

Exams / MLA-C02 / Evaluating ML and GenAI models / Question 1

MLA-C02 practice question: Evaluating ML and GenAI models

ML Model and Foundation Model (FM) Development A free question from our original MLA-C02 practice exam. Try it, then open the answer to see why each option is right or wrong.

Last verified against current AWS documentation ·How we write our questions

A public library is testing a Bedrock model to write short summaries of new book descriptions for its catalog. The team has 3,000 descriptions with reference summaries, and it wants to score the model's summaries for quality on a large scale.

Human reviewers cannot read all 3,000 summaries each week, and the team wants scores that judge meaning and coverage, not just word overlap. Results must be stored in an S3 bucket the team controls.

Which approach should the team take?

Choose one answer.

  1. A. Count how many words in each generated summary also appear in the reference summary, and report the average overlap.
  2. B. Send each summary to an Amazon Augmented AI human review loop with a team of reviewers, and score the model only after they finish all 3,000 items.
  3. C. Measure the average number of tokens in each summary and treat the shorter summaries as the better ones.
  4. D. Run a Bedrock model evaluation job that uses a judge model to score the responses to the S3 prompt dataset on the chosen quality metrics.
Show the answer and explanation

Answer: D

  • A. Wrong: Word overlap ignores meaning and rewording, so a good paraphrase can score poorly. The stem asks for judgments of quality.
  • B. Wrong: Human review is accurate but slow and costly at this volume each week. The stem asks for scoring at a large scale without reading all items by hand.
  • C. Wrong: Length does not show accuracy or coverage, and a short summary can miss the key points. The metric does not judge quality.
  • D. Right: Correct. A model evaluation job that uses a judge model sends the prompts in the dataset to the model, and a second LLM scores each response on the selected metrics and explains its score. The report is written to the S3 bucket the team specifies. Learn more: Evaluate model performance using another LLM as a judge (opens in a new tab)

Explanation

A Bedrock model evaluation job with a judge model runs a custom prompt dataset through the model, and a second LLM scores each response on quality metrics, with the report written to S3. Word overlap misses paraphrase, human review does not scale to 3,000 items each week, and token length does not measure quality.

Study

Keep practicing

Answer questions with a score at the end, or practice just this topic with the explanation after each one. No sign-up needed.

Take the free MLA-C02 practice examPractice Evaluating ML and GenAI modelsAll Evaluating ML and GenAI models questions

Want more on Evaluating ML and GenAI models? The 5 full-length practice sets have 6 more Evaluating ML and GenAI models questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets