AIP-C01 practice question: Quality gates and validation for model and prompt updates
Testing, Validation, and Troubleshooting A free question from our original AIP-C01 practice exam. Try it, then open the answer to see why each option is right or wrong.
A game studio uses a large model to write dialogue for non-player characters. The studio is moving to a newer model version, and it also keeps a separate prompt version for each character's voice. Prompt versions live in Amazon Bedrock Prompt Management, and production reads the version to use from a Parameter Store parameter.
A regression in one category, such as a rude remark slipping into a children's quest, is much more damaging than a small dip in fluency. The studio's evaluation suite scores each reply on five categories: accuracy, fluency, tone, safety and lore consistency. The studio also needs to roll back in seconds if production gets worse.
Which promotion process meets these requirements?
Choose one answer.
AA. Compare only the average score across all five categories, and move the production parameter to the new version whenever the average is equal to or higher than the current version's average.
BB. Move the production parameter to the new version first, then run the evaluation suite overnight and revert the parameter the next morning if the scores have dropped.
CC. Have a designer read a sample of 20 new dialogue lines before each release, then promote the new version by hand in the console if the lines look fine.
DD. Run the full suite on the candidate in the pipeline, require every category to meet its own threshold, and move the production parameter only on a pass.
Show the answer and explanation
Answer: D
A. Wrong: An average can hide a drop in safety when fluency rises, so a harmful regression could pass the gate. Promotion would be safe on paper but wrong for the most damaging category.
B. Wrong: Players would see the regression for hours before a revert, which fails the damage requirement. Scoring after promotion is monitoring, not a quality gate.
C. Wrong: Twenty lines cannot show a rare safety regression across thousands of replies. Manual promotion is also slow and error-prone for a studio that updates often.
Per-category thresholds catch regressions that an overall average would hide, and the pipeline moves the production parameter only after a pass. Keeping the previous prompt version lets the team point the parameter back in seconds. Averages, post-release checks and manual sampling each let a damaging regression reach players or depend on people to catch it.
Want more on Quality gates and validation for model and prompt updates? The 5 full-length practice sets have 5 more Quality gates and validation for model and prompt updates questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →