ExamPark
Course menuMLA-C02

Exams / MLA-C02 / Study guide / Evaluating ML and GenAI models

Evaluating ML and GenAI models: MLA-C02 study notes and practice questions

What to know about Evaluating ML and GenAI models for the MLA-C02 exam, from the ML Model and Foundation Model (FM) Development domain (24% of the exam guide), then 1 free question to test it. Part of the MLA-C02 study guide.

Last verified against current AWS documentation ·How we write our questions

What to know for the exam

Evaluation questions ask which metric to use, how to read a confusion matrix, or how to compare generative AI outputs.

Classification

Metric Means Prefer when
Precision Of predicted positives, how many were right False positives are costly (flagging good customers)
Recall Of real positives, how many were found False negatives are costly (missing fraud or disease)
F1 Balance of precision and recall Imbalanced classes, both errors matter
ROC AUC Ranking quality across thresholds Comparing models independent of threshold
PR AUC Precision-recall trade-off Very rare positives
Accuracy Share correct Only for balanced classes

A confusion matrix shows true and false positives and negatives; change the threshold to move between precision and recall.

Regression

  • RMSE penalizes large errors more; MAE treats all errors evenly and resists outliers.
  • R² shows the share of variance explained; MAPE gives percentage error.

Generative AI in Bedrock

  • Automatic evaluation with built-in datasets or your own prompts and metrics such as accuracy, robustness and toxicity.
  • LLM as a judge: a second model scores responses on correctness, completeness, helpfulness and harmfulness, at scale.
  • Human evaluation with your own team or an AWS-managed workforce, for subjective quality.
  • RAG evaluation scores retrieval (context relevance and coverage) and generation (correctness, faithfulness) for knowledge bases.
  • Reports are written to S3.
  • Text overlap metrics (ROUGE, BLEU, BERTScore) compare outputs with references but miss paraphrases.

Comparing in production

  • Shadow tests send copies of live traffic to a new model without affecting users.
  • A/B tests with production variants split traffic and compare business metrics.

Common traps

  • Accuracy on imbalanced data.
  • ROUGE alone to judge open-ended answers.
  • Human review of thousands of responses when an LLM judge scales.

Try the 1 practice question

Practice Evaluating ML and GenAI modelsTake the free practice exam

Question 1

Medium ·Link to this question (question 1)

A public library is testing a Bedrock model to write short summaries of new book descriptions for its catalog. The team has 3,000 descriptions with reference summaries, and it wants to score the model's summaries for quality on a large scale.

Human reviewers cannot read all 3,000 summaries each week, and the team wants scores that judge meaning and coverage, not just word overlap. Results must be stored in an S3 bucket the team controls.

Which approach should the team take?

Choose one answer.

  1. A. Count how many words in each generated summary also appear in the reference summary, and report the average overlap.
  2. B. Send each summary to an Amazon Augmented AI human review loop with a team of reviewers, and score the model only after they finish all 3,000 items.
  3. C. Measure the average number of tokens in each summary and treat the shorter summaries as the better ones.
  4. D. Run a Bedrock model evaluation job that uses a judge model to score the responses to the S3 prompt dataset on the chosen quality metrics.
Show the answer and explanation

Answer: D

  • A. Wrong: Word overlap ignores meaning and rewording, so a good paraphrase can score poorly. The stem asks for judgments of quality.
  • B. Wrong: Human review is accurate but slow and costly at this volume each week. The stem asks for scoring at a large scale without reading all items by hand.
  • C. Wrong: Length does not show accuracy or coverage, and a short summary can miss the key points. The metric does not judge quality.
  • D. Right: Correct. A model evaluation job that uses a judge model sends the prompts in the dataset to the model, and a second LLM scores each response on the selected metrics and explains its score. The report is written to the S3 bucket the team specifies. Learn more: Evaluate model performance using another LLM as a judge (opens in a new tab)

Explanation

A Bedrock model evaluation job with a judge model runs a custom prompt dataset through the model, and a second LLM scores each response on quality metrics, with the report written to S3. Word overlap misses paraphrase, human review does not scale to 3,000 items each week, and token length does not measure quality.

Study

Want more on Evaluating ML and GenAI models? The 5 full-length practice sets have 6 more Evaluating ML and GenAI models questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets

Study Evaluating ML and GenAI models

More ML Model and Foundation Model (FM) Development topics