Evaluating foundation models: AIF-C01 study notes and practice questions
What to know about Evaluating foundation models for the AIF-C01 exam, from the Applications of Foundation Models domain (28% of the exam guide), then 3 free questions to test it. Part of the AIF-C01 study guide.
Evaluation questions ask how to tell whether a model or application is good enough, and which metric fits a task.
Ways to evaluate
Method
Strength
Human evaluation
Judges qualities like helpfulness, tone and correctness that metrics miss; slower and costlier
Automatic metrics
Fast and repeatable on many outputs
Benchmark datasets
Standard tests to compare models on known tasks
Model as a judge
A model scores responses against criteria, practical at large scale
A common mix: automatic evaluation for every output, plus human review of a sample.
Metrics
Metric
Measures
Used for
ROUGE
Overlap between generated text and reference text
Summarization
BLEU
Overlap of word sequences with reference translations
Translation
BERTScore
Similarity of meaning, using embeddings
Any generated text where wording can vary
Accuracy / F1
Correct classifications
Classification tasks
Perplexity
How well a model predicts text
Language model quality
On AWS
Amazon Bedrock evaluations run automatic evaluation (with built-in or your own datasets), human evaluation with your team or AWS-managed workers, model-as-a-judge evaluation, and evaluation of RAG knowledge bases for retrieval and answer quality.
Did it meet the goal?
Technical scores aren’t the end. Check business outcomes (conversion, satisfaction, time saved), cost per request, latency, safety (toxicity, bias) and robustness to unusual inputs.
Common traps
ROUGE vs BLEU. ROUGE for summaries, BLEU for translation.
Only automatic scores. They miss tone and subtle errors; add human review.
Testing on easy examples. Include real, messy and adversarial inputs.
Evaluating once. Re-run the same evaluation whenever you change the model, the prompt or the retrieved data, so you can compare versions fairly.
B. Wrong: A model judging its own output is not an independent check. It also ignores the reference translations the publisher already has, so the result does not show which model is more accurate.
C. Wrong: Full human review is thorough but far beyond the review budget, which covers only a small sample. It is the wrong approach for a choice at this scale.
D. Wrong: Word count says nothing about whether the German keeps the meaning of the English. Two translations of the same length can differ greatly in quality.
Explanation
Automatic evaluation against reference data gives a repeatable score for every output, and a sample of human review checks what automated scores can miss. Both fit the scale and the budget in this scenario. This question tests choosing an evaluation method that matches the volume and the review resources.
A support team has 5,000 chatbot answers from last month.
It wants each answer scored for correctness and completeness, applied consistently, without reviewers reading every answer by hand. The team plans to run the evaluation on Amazon Bedrock.
Which approach fits BEST?
Choose one answer.
AA. Create a Bedrock model evaluation job with a judge model and the built-in Correctness and Completeness metrics.
BB. Compute a BLEU score for each answer against a stored reference answer, then sort all the answers by that score.
CC. Ask reviewers to read a random 50 answers and apply their average score to the full set of 5,000 answers.
DD. Count the thumbs-up and thumbs-down clicks customers gave each answer, and rank the answers by total.
B. Wrong: BLEU measures word overlap with a reference. It does not check whether the facts are correct or whether the answer covers everything the customer needed.
C. Wrong: A sample gives only an estimate for the whole set, and the team wants a score for every answer. Reviewers would still be reading many answers by hand.
D. Wrong: Clicks reflect how users felt, not whether the answer is correct or complete. Many answers will also have no votes at all.
Explanation
Judge-model evaluation on Amazon Bedrock uses a model to score responses against chosen metrics, which makes large-scale quality checks practical. Overlap scores and customer clicks measure different things. This question tests choosing an evaluation method that matches both the criteria and the volume.
An online retailer launched a generative AI shopping assistant three months ago. The board asks whether it is delivering business value and wants one measure that links the assistant to sales.
Which measure BEST answers that question?
Choose one answer.
AA. Conversion rate of chat users to orders, compared with a control group of shoppers who never used the assistant.
BB. Perplexity score of the model on a held-out set of product questions, which measures how well it predicts text.
CC. Average number of tokens the assistant generates per reply, which drives the model's per-request usage cost.
DD. Number of prompt revisions the team made to the assistant's wording during the three months after launch.
Show the answer and explanation
Answer: A
A. Right: Correct. Comparing orders from chat users with a control group links the assistant to revenue, which is the business value the board asks about.
B. Wrong: Perplexity measures how well a language model predicts text. It says nothing about whether shoppers buy, so it does not answer the board's question.
C. Wrong: Tokens per reply affect cost and speed, which are operational concerns. They do not show whether the assistant brings in sales.
D. Wrong: Prompt revisions measure team effort, not results. Heavy editing could still leave the assistant with no effect on sales.
Explanation
Business value for a generative AI assistant is judged by outcomes the business cares about, such as sales and conversion, not by model internals or team activity. A control group helps show that the assistant, rather than other factors, caused any change. This question tests choosing a business measure over technical ones.
Want more on Evaluating foundation models? The 5 full-length practice sets have 21 more Evaluating foundation models questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →