Human review and approval in AI workflows: AIP-C01 study notes and practice questions
What to know about Human review and approval in AI workflows for the AIP-C01 exam, from the Implementation and Integration domain (26% of the exam guide), then 1 free question to test it. Part of the AIP-C01 study guide.
AIP-C01 describes an AI workflow that needs a person at some point and asks how to wire that person in without blocking the whole system. Amazon SageMaker AI Augmented AI (A2I) is closed to new customers, so build human review with AWS Step Functions task tokens, Amazon API Gateway and your own review queue.
Pausing for approval
Step Functions supports the task token callback pattern (.waitForTaskToken). The state machine sends a token with the review request, such as to an SQS queue or a review application, then pauses. The reviewer’s app returns the token with SendTaskSuccess or SendTaskFailure, and the workflow continues. The wait can last hours or days without polling. Pair the wait with a timeout.
Routing by confidence
Send an output to a reviewer only when its confidence is below a threshold, and let confident outputs pass. Set the threshold on labeled test data, because a raw score is not always a calibrated probability. Track how many outputs go to review, since a threshold that is too low floods reviewers.
Feedback and corrections
Expose an API Gateway endpoint where users or reviewers flag bad answers or submit corrections. Store the feedback, and add the corrected examples to the evaluation dataset. That way each correction becomes a regression test for the next prompt or model version.
Approval before external action
When an agent is about to send an email or move money, have it prepare the action and wait for a human approval step before the tool runs. The approval gate belongs in the workflow, not in the prompt.
Escalation on timeout
If no reviewer responds within the allowed time, escalate to a backup queue or a senior reviewer. Use a Step Functions Catch on the timeout. Fail safe: a timed-out request is not approved automatically.
How to choose
Scenario wording
Choose
“Pause a workflow until a person approves, hours later”
Step Functions task token callback
“Only uncertain outputs need a person”
Confidence threshold routing to a review queue
“Users report wrong answers”
API Gateway feedback endpoint with stored feedback
“Improve the next prompt or model from reviewer edits”
Add corrections to the evaluation dataset
“Agent is about to send a payment or email”
Approval gate before the tool call
“Reviewer does not respond”
Timeout with escalation, never automatic approval
Common traps
SageMaker AI Augmented AI for a new build. It is closed to new customers.
Polling a database in a loop to see whether a reviewer finished, instead of a task token callback.
Auto-approving on timeout to keep the workflow moving.
Reviewer corrections that never reach evaluation, so the same errors come back.
Treating a raw model confidence score as a reliable probability without testing it on labeled data.
Sending every output to review, which removes the value of automation.
A bank's loan operations team uses a generative AI workflow to draft adverse-action letters for declined loan applications. Each draft must be approved or rejected by a credit officer in an internal web app before any letter is sent.
Officers respond anywhere from ten minutes to two days after a draft is created. The workflow must not keep a Lambda function or other paid compute running while it waits for a decision.
No letter may be sent without a recorded approval, and a rejected draft must be regenerated with the officer's comments added to the prompt. The team wants the least custom code to maintain.
Which design meets these requirements?
Choose one answer.
AA. Run a Lambda function that checks a DynamoDB table every 30 seconds for an approval record, and keeps checking for up to two days before it sends or discards the draft.
BB. Email each draft to the officer through Amazon SES, and have the workflow send the letter automatically after 24 hours unless the officer rejects it in the web app.
CC. Have the officer's approval button in the web app call the send-letter Lambda function directly, and skip recording the decision in the workflow so that no state has to be stored.
DD. Use a Step Functions Standard workflow task with .waitForTaskToken that saves the token with the draft, and have the web app call SendTaskSuccess with the decision.
Show the answer and explanation
Answer: D
A. Wrong: A Lambda function cannot run for two days, and a polling loop would hold compute open and bill for every check. The design is also costly to maintain, which conflicts with the requirement for the least custom code.
B. Wrong: Sending a letter after a timeout means a letter can go out with no recorded approval, which breaks the rule. The design also does not regenerate drafts with officer comments on rejection.
C. Wrong: Calling the sender directly removes the approval record that the audit trail depends on, and the workflow has no way to regenerate a rejected draft. It also mixes approval logic into the web app.
D. Right: A task that waits for a callback holds no compute while it waits, and a Standard workflow can pause for days. The callback records each decision, and a Choice state then routes approved drafts to sending and rejected drafts back to generation with the comments. Learn more: Wait for a callback with a task token (opens in a new tab)
Explanation
Step Functions can pause a workflow at a task that waits for a callback, and a task token lets the web app resume it with the officer's decision. The Choice state then sends only approved letters and regenerates rejected drafts with the comments. Polling, timed auto-send and direct calls each break the waiting, approval or regeneration rule.
Want more on Human review and approval in AI workflows? The 5 full-length practice sets have 5 more Human review and approval in AI workflows questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →