Troubleshooting prompts and inconsistent responses: AIP-C01 study notes and practice questions
What to know about Troubleshooting prompts and inconsistent responses for the AIP-C01 exam, from the Testing, Validation, and Troubleshooting domain (11% of the exam guide), then 1 free question to test it. Part of the AIP-C01 study guide.
Start by proving the problem with data. Find which prompt version, input or retrieved context changed the output before you rewrite the prompt.
Format inconsistency
When the output shape varies, add an explicit output schema, such as a JSON schema with required fields, and two or three examples of correct output that include edge cases. Structured outputs enforce the schema on models that support them. For other models, validate the output in code, and retry once with the validation error added to the prompt.
Instructions lost in long context
A model may skip a rule buried deep in a long prompt or placed after a large document. Move critical rules to the system prompt or to the end of the prompt, state each rule once and clearly, and remove context that does not help the task. Test the change on the same prompts, because moving text can fix one case and break another.
Comparing prompt versions
Use Amazon Bedrock Prompt Management to version templates and variants, so each output traces to a prompt version. Run the old and new versions over the same golden dataset, and compare pass rate, format compliance and hallucination rate. Output diffing shows exactly which answers changed, so review those first.
Finding hallucination increases
Keep a golden dataset with expected answers and sources. When hallucinations rise, decide whether the cause is the prompt, the retrieved context or the model. Run the same prompt with fixed retrieved context to isolate the prompt. Logs that record prompt versions, plus reviewer notes, often show the pattern, such as a new example that encourages guessing.
Conflicting instructions
Two instructions that pull in opposite directions, such as “be brief” and “include every detail”, cause inconsistent answers. Search the logs for the inputs that produced bad outputs, find the contradictory line, and rewrite one of the two rules.
How to choose
Scenario wording
Choose
Output shape varies between calls
JSON schema, examples, structured outputs
The model ignores a rule in a long prompt
Move the rule to the system prompt or the end, and cut context
Quality dropped after a template edit
Compare versions on the golden dataset, then diff outputs
Hallucinations are rising
Golden set with sources, tested with fixed context
Two instructions contradict each other
Find them in the logs and rewrite one rule
Need to see what changed between runs
Output diffing across runs
Common traps
Rewriting the whole prompt when one conflicting line causes the failure.
Adding more instructions when the real problem is format, which examples fix better.
Testing on a few hand-picked inputs, not the golden set.
Blaming the model when retrieval supplied the wrong context.
Not recording the prompt version, so a regression cannot be traced.
An HR analytics team asks a model to read job requisition text and return a JSON record for a hiring dashboard. The dashboard's loader expects exact key names such as title, department and openDate. In some replies the model adds a sentence of explanation before the JSON, renames a key, or leaves a trailing comma, and the loader then rejects the record.
The team needs parsing failures below half a percent, the same key names every time, and no changes to the loader. The responses should be as repeatable as possible from run to run.
Which change meets these requirements with the LEAST code change?
Choose one answer.
AA. State the exact keys and value types in the prompt, add two worked examples of requisition text with correct JSON output, and set a low temperature.
BB. Raise the temperature to 1.0 so that the model explores more varied answers, which helps it recover from requisitions with unusual wording.
CC. Keep the prompt as it is, and add a second model call that rewrites each reply into valid JSON, so that the loader always receives a clean record.
DD. Switch to a larger model in the same family, because larger models follow formatting instructions more closely and need no changes to the prompt.
Show the answer and explanation
Answer: A
A. Right: Explicit keys, types and examples steer the model to one format, and a low temperature makes replies more repeatable. The loader needs no change, and the prompt is the only file the team edits.
B. Wrong: A higher temperature makes replies less repeatable, which increases format drift. It works against the same-keys requirement.
C. Wrong: A rewrite call can fix formatting, but it doubles the model cost and latency for every record. The key names still depend on the model's first reply, so the design is not reliable.
D. Wrong: A larger model may follow instructions better, but the team gets no guarantee of exact keys and costs more per record. The prompt would still lack the explicit format the team needs.
Explanation
Format drift is reduced by stating the exact keys and types, showing worked examples, and lowering the temperature so replies repeat. This keeps the loader unchanged and the edit small. A higher temperature, a rewrite pass or a bigger model each add variation, cost or latency without fixing the format.
Want more on Troubleshooting prompts and inconsistent responses? The 5 full-length practice sets have 5 more Troubleshooting prompts and inconsistent responses questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →
Study Troubleshooting prompts and inconsistent responses