AIP-C01 practice question: Testing prompts for regressions and edge cases
Foundation Model Integration, Data Management, and Compliance A free question from our original AIP-C01 practice exam. Try it, then open the answer to see why each option is right or wrong.
A recipe website's content team edits its generation prompt several times a week. Last month, a small wording change made the model drop the allergen section from some recipes, and readers found out before the team did.
The team wants to catch this kind of regression before a change reaches readers. Its developers already run tests in a CI pipeline for every code commit.
The check should take little time to run and should fail the pipeline when quality drops.
Which approach meets these requirements?
Choose one answer.
AA. Read a sample of live generated recipes once a month, and ask the content team to report any problems they notice before the next prompt change is released.
BB. Turn on model invocation logging, and have an analyst review a random set of logged responses each week to look for missing sections.
CC. Keep a golden set of recipe inputs with the sections each output must contain, and run it in CI on every prompt change, failing the build below a pass threshold.
DD. Switch every recipe request to the largest model in the family, since a stronger model drops sections less often, and stop testing prompt changes.
Show the answer and explanation
Answer: C
A. Wrong: Monthly sampling finds regressions weeks after readers see them, which is the problem the team is trying to solve. Nothing in the pipeline would stop a bad prompt from being released.
B. Wrong: Invocation logs show what happened after a change went live, so problems are still found too late. The review is also manual and does not block a release.
C. Right: A golden set gives a fixed, repeatable check that runs on every change, so a missing allergen section fails the build before release. It is quick to run and it blocks a bad change, which meets all the stated requirements.
D. Wrong: A stronger model may reduce some errors, but it does not guarantee that a wording change keeps every required section. Removing the tests also removes the only check the team has, so the approach misses the goal.
Explanation
A golden test set with required sections, run in CI for every prompt change, catches regressions before release and fails the build when quality drops. Sampling after release, log review and relying on a bigger model all find problems too late or not at all.
Want more on Testing prompts for regressions and edge cases? The 5 full-length practice sets have 6 more Testing prompts for regressions and edge cases questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →