ExamPark
Course menuAIP-C01

Exams / AIP-C01 / Fairness and bias evaluation / Question 1

AIP-C01 practice question: Fairness and bias evaluation

AI Safety, Security, and Governance A free question from our original AIP-C01 practice exam. Try it, then open the answer to see why each option is right or wrong.

Last verified against current AWS documentation ·How we write our questions

A staffing agency is about to release a model-based tool that drafts short candidate summaries for recruiters. The summaries are built from résumés and interview notes.

The agency's ethics review asks whether the drafts describe equally qualified candidates in the same way, regardless of name, gender or age. A single overall accuracy score on a general benchmark will not answer that, because differences can hide in a few subgroups.

The team must produce evidence before launch, using the model it will actually run, and must be able to repeat the same test after each prompt change.

Which approach meets these requirements?

Choose one answer.

  1. A. Build a paired test set in which each candidate profile appears twice with only the name, gender or age changed, run both versions through the model, and compare the scores.
  2. B. Run the general text quality benchmark on the model, and publish its overall accuracy score, which the vendor reports as high, as the evidence that the summaries are fair to candidates.
  3. C. Add a word filter that blocks gendered pronouns and age-related words in every summary, so the output contains no such terms, and record the filter as the fairness control for the review.
  4. D. Ask the model in the system prompt to treat all candidates equally, and have a recruiter read one summary per week to confirm that no bias is visible.
Show the answer and explanation

Answer: A

  • A. Right: Counterfactual pairs isolate the effect of one attribute, so a consistent gap in the summaries shows up as a measurable difference. The same paired set can be rerun after each prompt change to track the result before launch. Learn more: Evaluate model performance with an LLM as a judge (opens in a new tab)
  • B. Wrong: An overall score averages across groups and can hide a gap for one subgroup. It measures general quality, not whether equally qualified candidates are described the same way.
  • C. Wrong: Removing specific words does not stop bias that appears in other wording, such as phrasing about career gaps or school names. The check measures the filter, not how the summaries treat candidates. Learn more: Remove a specific list of words and phrases from conversations with word filters (opens in a new tab)
  • D. Wrong: A prompt instruction is a request, not evidence, and reading a few summaries each week cannot reveal small, consistent gaps. The option gives no repeatable measurement for the ethics review.

Explanation

Paired counterfactual prompts change only the attribute being tested, so any consistent difference in the summaries becomes measurable, and the same set can be rerun after each change. An overall benchmark score can hide subgroup gaps, and word filters, prompt instructions and occasional reading do not measure fairness.

Study

Keep practicing

Answer questions with a score at the end, or practice just this topic with the explanation after each one. No sign-up needed.

Take the free AIP-C01 practice examPractice Fairness and bias evaluationAll Fairness and bias evaluation questions

Want more on Fairness and bias evaluation? The 5 full-length practice sets have 7 more Fairness and bias evaluation questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets