ExamPark
Course menuAIP-C01

Exams / AIP-C01 / Evaluating model output quality (Bedrock evaluations, LLM-as-a-judge) / Question 1

AIP-C01 practice question: Evaluating model output quality (Bedrock evaluations, LLM-as-a-judge)

Testing, Validation, and Troubleshooting A free question from our original AIP-C01 practice exam. Try it, then open the answer to see why each option is right or wrong.

Last verified against current AWS documentation ·How we write our questions

A consumer electronics company's support team is choosing between two foundation models for a product FAQ assistant. The team has 500 real customer questions, each with a reference answer written by the support lead, stored as a JSONL file in Amazon S3.

The team has no machine learning engineers and only one week. It needs accuracy and toxicity results for each model, with no custom scoring code and no paid human reviewers. The results must be saved in Amazon S3 so that the procurement team can compare them in a report.

Which TWO actions meet these requirements? (Choose two.)

Choose two answers.

  1. A. Create a human evaluation job with a contract review team that rates 500 answers from each model on a five-point scale, and export the ratings to a spreadsheet.
  2. B. Write an AWS Lambda function that compares each model answer with its reference answer using exact text matching, and writes the scores to a CSV file in S3.
  3. C. Create an automatic model evaluation job with the built-in accuracy and toxicity metrics, using the team's JSONL file that includes the reference answers as the prompt dataset.
  4. D. Create an automatic model evaluation job with the built-in toxicity metric on a built-in prompt dataset, since those general prompts cover the same kinds of topics as customer questions.
  5. E. Run one automatic evaluation job for each candidate model, using the same JSONL dataset and metrics, and compare the result files that each job writes to S3.
Show the answer and explanation

Answers: C, E

  • A. Wrong: Human reviewers can judge quality well, but this design adds paid reviewers and extra days that the team does not have. The requirement rules out paid human reviewers.
  • B. Wrong: Exact matching misses correct answers worded differently, and the team wants no custom scoring code. Toxicity would also not be measured at all.
  • C. Right: The built-in metrics need no scoring code, and the job uses the team's own questions with their reference answers. Results are written to the S3 location the team chooses, which feeds the procurement report. Learn more: Create a model evaluation job using built-in metrics (opens in a new tab)
  • D. Wrong: The toxicity metric is the right one, but the built-in prompts are not the team's customer questions. Accuracy against the reference answers would also not be measured.
  • E. Right: Each automatic job evaluates a single model, so one job per model gives a like-for-like comparison on the same data. The result files in S3 are the inputs the procurement report needs. Learn more: Evaluate the performance of Amazon Bedrock resources (opens in a new tab)

Explanation

Automatic model evaluation jobs with built-in metrics score a model on the team's own dataset without custom code or paid reviewers. Running one job per candidate on the same dataset gives a fair comparison, and the S3 results feed the report. A human review panel, custom exact matching and a general prompt dataset each miss a stated requirement.

Study

Keep practicing

Answer questions with a score at the end, or practice just this topic with the explanation after each one. No sign-up needed.

Take the free AIP-C01 practice examPractice Evaluating model output quality (Bedrock evaluations, LLM-as-a-judge)All Evaluating model output quality (Bedrock evaluations, LLM-as-a-judge) questions

Want more on Evaluating model output quality (Bedrock evaluations, LLM-as-a-judge)? The 5 full-length practice sets have 5 more Evaluating model output quality (Bedrock evaluations, LLM-as-a-judge) questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets