Exams / AIP-C01 / Study guide / Evaluating model output quality (Bedrock evaluations, LLM-as-a-judge)

Evaluating model output quality (Bedrock evaluations, LLM-as-a-judge): AIP-C01 study notes and practice questions

What to know about Evaluating model output quality (Bedrock evaluations, LLM-as-a-judge) for the AIP-C01 exam, from the Testing, Validation, and Troubleshooting domain (11% of the exam guide), then 1 free question to test it. Part of the AIP-C01 study guide.

Last verified November 2026 against current AWS documentation ·How we write our questions

What to know for the exam

The exam expects you to match the evaluation method to the metric. Objective checks can be automated. Subjective quality needs a judge model or people, and the judge itself must be checked.

Automatic evaluation

A Bedrock automatic model evaluation job runs a model over a prompt dataset, either one you supply or a built-in dataset, and scores the responses with built-in metrics such as accuracy, robustness and toxicity. Use it for fast, repeatable comparisons when a standard metric applies or the expected output is known. Results are saved to S3.

LLM-as-a-judge

A judge model scores each response against a rubric, such as helpfulness, correctness or harmfulness. Bedrock supports built-in quality and responsible AI metrics, and custom metrics that you define in plain language with your own scoring scale. Validate the judge before you trust it: compare its scores with a sample of answers that people have labeled.

Human evaluation

A human evaluation job sends responses to your own work team, who rate them against your criteria. Use it for subjective qualities such as tone or domain correctness that a judge may miss. It is slower and costs more, so use it to calibrate the judge and for high-risk releases.

Compare models on the full picture

Run the same prompt dataset through each candidate, then compare quality with cost per task and latency percentiles. The best model is the cheapest one that passes the quality threshold on your data, not the one with the highest average score. Repeat the comparison when a provider releases a new version.

How to choose

Scenario wording	Choose
Known correct answers and a repeatable score	Automatic evaluation with built-in metrics
Score helpfulness against our own rubric	LLM-as-a-judge with a custom metric
Subjective quality that needs people	Human evaluation job with your work team
Pick between two models on cost and quality	Same dataset, compared on quality, cost and latency
Answers grounded in documents	RAG evaluation
Test a candidate on live traffic	Canary release with quality gates

Common traps

Trusting a judge without validating it against human labels.
Picking the highest quality score while ignoring the cost and latency the question gives.
Testing on public benchmarks instead of representative prompts from your application.
Running a human job for every release when automatic checks answer the question.
Evaluating on the same prompts used for tuning, which overstates the results.

Try the 1 practice question

Practice Evaluating model output quality (Bedrock evaluations, LLM-as-a-judge)Take the free practice exam

Question 1

Harder · choose 2 ·Link to this question

A consumer electronics company's support team is choosing between two foundation models for a product FAQ assistant. The team has 500 real customer questions, each with a reference answer written by the support lead, stored as a JSONL file in Amazon S3.

The team has no machine learning engineers and only one week. It needs accuracy and toxicity results for each model, with no custom scoring code and no paid human reviewers. The results must be saved in Amazon S3 so that the procurement team can compare them in a report.

Which TWO actions meet these requirements? (Choose two.)

Choose two answers.

A. Create a human evaluation job with a contract review team that rates 500 answers from each model on a five-point scale, and export the ratings to a spreadsheet.
B. Write an AWS Lambda function that compares each model answer with its reference answer using exact text matching, and writes the scores to a CSV file in S3.
C. Create an automatic model evaluation job with the built-in accuracy and toxicity metrics, using the team's JSONL file that includes the reference answers as the prompt dataset.
D. Create an automatic model evaluation job with the built-in toxicity metric on a built-in prompt dataset, since those general prompts cover the same kinds of topics as customer questions.
E. Run one automatic evaluation job for each candidate model, using the same JSONL dataset and metrics, and compare the result files that each job writes to S3.

Show the answer and explanation

Answers: C, E

A. Wrong: Human reviewers can judge quality well, but this design adds paid reviewers and extra days that the team does not have. The requirement rules out paid human reviewers.
B. Wrong: Exact matching misses correct answers worded differently, and the team wants no custom scoring code. Toxicity would also not be measured at all.
C. Right: The built-in metrics need no scoring code, and the job uses the team's own questions with their reference answers. Results are written to the S3 location the team chooses, which feeds the procurement report. Learn more: Create a model evaluation job using built-in metrics (opens in a new tab)
D. Wrong: The toxicity metric is the right one, but the built-in prompts are not the team's customer questions. Accuracy against the reference answers would also not be measured.
E. Right: Each automatic job evaluates a single model, so one job per model gives a like-for-like comparison on the same data. The result files in S3 are the inputs the procurement report needs. Learn more: Evaluate the performance of Amazon Bedrock resources (opens in a new tab)

Explanation

Automatic model evaluation jobs with built-in metrics score a model on the team's own dataset without custom code or paid reviewers. Running one job per candidate on the same dataset gives a fair comparison, and the S3 results feed the report. A human review panel, custom exact matching and a general prompt dataset each miss a stated requirement.

Study

Want more on Evaluating model output quality (Bedrock evaluations, LLM-as-a-judge)? The 5 full-length practice sets have 5 more Evaluating model output quality (Bedrock evaluations, LLM-as-a-judge) questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets

Study Evaluating model output quality (Bedrock evaluations, LLM-as-a-judge)

Evaluate the performance of Amazon Bedrock resources