AI Safety, Security, and Governance A free question from our original AIP-C01 practice exam. Try it, then open the answer to see why each option is right or wrong.
A telecom operator runs a support assistant that answers customers' questions from its internal device manuals. It retrieves passages from a knowledge base and passes them to a model on Amazon Bedrock.
Engineers have found that some answers add steps that do not appear in the retrieved passages, and a wrong router reset procedure could cause an outage. The operator wants any answer that is not supported by the retrieved passages to be stopped, while answers that restate the passages correctly should pass.
The team must tune the threshold on real traffic before it blocks anything, and the control must run on each answer without retraining the model.
Which solution meets these requirements?
Choose one answer.
AA. Set the model's temperature to 0 and add a system prompt that tells the model to answer only from the provided passages, relying on those instructions to prevent unsupported steps.
BB. Fine-tune the model on the device manuals so that it learns the correct procedures, and then test a sample of answers each quarter to confirm they are accurate.
CC. Attach a guardrail with a contextual grounding check to the Converse calls, pass the retrieved passages as the grounding source, and run it in Detect mode first.
DD. Replace the knowledge base with a model that has a much larger context window, so the whole manual fits into every prompt and the model cannot invent steps it has not been shown.
Show the answer and explanation
Answer: C
A. Wrong: A temperature of 0 reduces variation, and the prompt asks for grounded answers, but neither one checks whether a specific answer is supported. The requirement to stop unsupported answers at runtime is not met.
B. Wrong: Fine-tuning can improve accuracy on common procedures, but it does not verify each answer against the passages at runtime. Every manual update would need a new training run, and a quarterly sample cannot stop a single wrong answer. Learn more: Customize your model (opens in a new tab)
C. Right: The grounding check scores each answer for grounding in the source passages and for relevance to the question, and it blocks answers that fall under the thresholds. Detect mode lets the team tune those thresholds on real traffic before it enforces them. Learn more: Contextual grounding checks (opens in a new tab)
D. Wrong: A larger context window reduces missed content, but the model can still add steps that are not in the manual, and sending the whole manual raises cost and latency. It gives no per-answer check that the operator can tune or audit.
Explanation
The contextual grounding check scores each answer against the retrieved passages and the user's question, and its grounding and relevance thresholds decide what gets blocked. Running it in Detect mode first lets the team tune those thresholds on real traffic. Prompt instructions, fine-tuning and larger context windows do not verify individual answers.
Want more on Reducing hallucinations (grounding, contextual grounding checks, Automated Reasoning, structured outputs)? The 5 full-length practice sets have 5 more Reducing hallucinations (grounding, contextual grounding checks, Automated Reasoning, structured outputs) questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →