Testing prompts for regressions and edge cases: AIP-C01 study notes and practice questions
What to know about Testing prompts for regressions and edge cases for the AIP-C01 exam, from the Foundation Model Integration, Data Management, and Compliance domain (31% of the exam guide), then 1 free question to test it. Part of the AIP-C01 study guide.
AIP-C01 asks how to prove a prompt change is safe, which test catches which failure, and how to detect a problem that appears only in production.
Test layers
Layer
What it checks
Tool
Golden test set
Answers to a fixed list of inputs with known good outputs
Bedrock evaluations or a test script in CI
Format checks
Output is valid JSON and has the required fields
Lambda function that parses and validates
Edge-case suite
Empty input, very long input, attacks, mixed languages
Step Functions workflow running many cases in parallel
A/B comparison
Two prompt versions on the same dataset
Prompt variants or an evaluation job, then compare scores
Production monitoring
Regressions after release
CloudWatch metrics and alarms
Run the golden set on every prompt change, and block the release when a metric falls below its threshold. Keep the same dataset across runs so scores are comparable.
Evaluation choices
Bedrock model evaluation offers automatic jobs with built-in or custom prompt datasets, LLM-as-a-judge scoring for quality and custom criteria, and human evaluation with your own work team. Use automatic metrics for speed and LLM-as-a-judge for qualities that are hard to check by rule. Use humans for final calibration. Amazon SageMaker AI Ground Truth and Augmented AI are closed to new customers, so use Bedrock human evaluation or Step Functions task tokens for human review.
Format drift
A model version change can alter output formatting even when the prompt is untouched. Detect it by running the format checks against the new model before switching traffic. Use Bedrock Flows or prompt versions to pin the old and new model choices side by side during the test.
How to choose
Scenario wording
Choose
“Every prompt edit must pass tests before deployment”
Golden test set run in CI with a release gate
“Downstream parser fails on some responses”
Lambda JSON schema validation
“Compare two wordings on the same questions”
A/B comparison with the same dataset
“Quality dropped after release”
CloudWatch metrics and alarms, then rollback
“Model upgrade changes output format”
Format checks on the new model before switching
Common traps
Testing one happy-path example and calling the prompt validated.
Changing the dataset between runs, which makes scores incomparable.
Trusting only LLM-as-a-judge for safety or exact-format checks, where rules are more reliable.
Reaching for SageMaker Clarify or Model Monitor for new work. Both are closed to new customers.
Testing the prompt without the model version it runs on.
A recipe website's content team edits its generation prompt several times a week. Last month, a small wording change made the model drop the allergen section from some recipes, and readers found out before the team did.
The team wants to catch this kind of regression before a change reaches readers. Its developers already run tests in a CI pipeline for every code commit.
The check should take little time to run and should fail the pipeline when quality drops.
Which approach meets these requirements?
Choose one answer.
AA. Read a sample of live generated recipes once a month, and ask the content team to report any problems they notice before the next prompt change is released.
BB. Turn on model invocation logging, and have an analyst review a random set of logged responses each week to look for missing sections.
CC. Keep a golden set of recipe inputs with the sections each output must contain, and run it in CI on every prompt change, failing the build below a pass threshold.
DD. Switch every recipe request to the largest model in the family, since a stronger model drops sections less often, and stop testing prompt changes.
Show the answer and explanation
Answer: C
A. Wrong: Monthly sampling finds regressions weeks after readers see them, which is the problem the team is trying to solve. Nothing in the pipeline would stop a bad prompt from being released.
B. Wrong: Invocation logs show what happened after a change went live, so problems are still found too late. The review is also manual and does not block a release.
C. Right: A golden set gives a fixed, repeatable check that runs on every change, so a missing allergen section fails the build before release. It is quick to run and it blocks a bad change, which meets all the stated requirements.
D. Wrong: A stronger model may reduce some errors, but it does not guarantee that a wording change keeps every required section. Removing the tests also removes the only check the team has, so the approach misses the goal.
Explanation
A golden test set with required sections, run in CI for every prompt change, catches regressions before release and fails the build when quality drops. Sampling after release, log review and relying on a bigger model all find problems too late or not at all.
Want more on Testing prompts for regressions and edge cases? The 5 full-length practice sets have 6 more Testing prompts for regressions and edge cases questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →
Study Testing prompts for regressions and edge cases