Reducing hallucinations (grounding, contextual grounding checks, Automated Reasoning, structured outputs): AIP-C01 study notes and practice questions
What to know about Reducing hallucinations (grounding, contextual grounding checks, Automated Reasoning, structured outputs) for the AIP-C01 exam, from the AI Safety, Security, and Governance domain (20% of the exam guide), then 1 free question to test it. Part of the AIP-C01 study guide.
AIP-C01 asks how to reduce invented facts in generative answers, and which control to use for each kind of claim. The answer depends on the source of truth: retrieved documents, written policy rules, or a required data shape.
Contextual grounding checks
A contextual grounding check scores a model response on two thresholds. The grounding score measures whether the answer is supported by the source passages. The relevance score measures whether it answers the user’s query. You set a threshold for each, and a response below either one is blocked or flagged. Pass the retrieved passages, the query and the response as tagged content so the check knows what to compare. Raise thresholds for regulated answers, and use lower thresholds for summaries where some paraphrase is expected. See contextual grounding checks.
Automated Reasoning checks
Automated Reasoning checks test a model answer against formal rules built from a policy document, such as a benefits or eligibility policy. The check reports whether the answer is consistent with those rules. It fits policies with exact conditions, such as numeric limits and yes-or-no eligibility. It does not verify general facts, so use grounding checks for those. See Automated Reasoning checks.
Structured outputs
Ask for JSON that matches a schema, so the model cannot add unexpected fields or return the wrong type. Validate the result in code. When validation fails, retry once or fall back to a safe response. Native structured outputs are offered only for some models. For others, put the schema in a tool definition or the prompt.
Citations and refusals
Require the answer to cite the passages it used. If retrieval returns nothing relevant, the correct behavior is to say the information was not found, not to generate a plausible answer. Put this rule in the prompt and test it with questions that have no answer in the corpus.
Verification and fallback
Compare the answer with its source using semantic similarity as a second check. Where the application has a confidence signal, route low-confidence answers to a person. A Step Functions workflow with a task-token callback can pause for review and then release the answer.
How to choose
Scenario wording
Choose
“Answers must be supported by retrieved passages”
Contextual grounding check with grounding and relevance thresholds
“Eligibility or pricing rules must be applied exactly”
Automated Reasoning checks against the written policy
“A downstream system needs fixed fields and types”
Structured output with a JSON schema and server-side validation
“No documents match the question”
Refuse with a not-found message, with no generated guess
“Low confidence answers must reach staff”
Confidence threshold with a human review callback
Common traps
Fine-tuning to fix factual errors when the facts change. Retrieval with grounding checks keeps answers current without retraining.
Believing temperature 0 removes hallucinations. Lower temperature reduces randomness but does not guarantee support.
Running a grounding check without passing the source passages, so there is nothing to compare against.
Using Automated Reasoning for open-ended content that has no written rules behind it.
Choosing a larger model as the main fix instead of adding evidence and checks.
A telecom operator runs a support assistant that answers customers' questions from its internal device manuals. It retrieves passages from a knowledge base and passes them to a model on Amazon Bedrock.
Engineers have found that some answers add steps that do not appear in the retrieved passages, and a wrong router reset procedure could cause an outage. The operator wants any answer that is not supported by the retrieved passages to be stopped, while answers that restate the passages correctly should pass.
The team must tune the threshold on real traffic before it blocks anything, and the control must run on each answer without retraining the model.
Which solution meets these requirements?
Choose one answer.
AA. Set the model's temperature to 0 and add a system prompt that tells the model to answer only from the provided passages, relying on those instructions to prevent unsupported steps.
BB. Fine-tune the model on the device manuals so that it learns the correct procedures, and then test a sample of answers each quarter to confirm they are accurate.
CC. Attach a guardrail with a contextual grounding check to the Converse calls, pass the retrieved passages as the grounding source, and run it in Detect mode first.
DD. Replace the knowledge base with a model that has a much larger context window, so the whole manual fits into every prompt and the model cannot invent steps it has not been shown.
Show the answer and explanation
Answer: C
A. Wrong: A temperature of 0 reduces variation, and the prompt asks for grounded answers, but neither one checks whether a specific answer is supported. The requirement to stop unsupported answers at runtime is not met.
B. Wrong: Fine-tuning can improve accuracy on common procedures, but it does not verify each answer against the passages at runtime. Every manual update would need a new training run, and a quarterly sample cannot stop a single wrong answer. Learn more: Customize your model (opens in a new tab)
C. Right: The grounding check scores each answer for grounding in the source passages and for relevance to the question, and it blocks answers that fall under the thresholds. Detect mode lets the team tune those thresholds on real traffic before it enforces them. Learn more: Contextual grounding checks (opens in a new tab)
D. Wrong: A larger context window reduces missed content, but the model can still add steps that are not in the manual, and sending the whole manual raises cost and latency. It gives no per-answer check that the operator can tune or audit.
Explanation
The contextual grounding check scores each answer against the retrieved passages and the user's question, and its grounding and relevance thresholds decide what gets blocked. Running it in Detect mode first lets the team tune those thresholds on real traffic. Prompt instructions, fine-tuning and larger context windows do not verify individual answers.
Want more on Reducing hallucinations (grounding, contextual grounding checks, Automated Reasoning, structured outputs)? The 5 full-length practice sets have 5 more Reducing hallucinations (grounding, contextual grounding checks, Automated Reasoning, structured outputs) questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →