User feedback, annotation and reporting: AIP-C01 study notes and practice questions
What to know about User feedback, annotation and reporting for the AIP-C01 exam, from the Testing, Validation, and Troubleshooting domain (11% of the exam guide), then 1 free question to test it. Part of the AIP-C01 study guide.
Feedback helps only when it links to the exact response, is rated consistently, and feeds back into testing. The exam tests each of those links.
Capture feedback with IDs
Record each thumbs up or down with the request ID, the model ID, the prompt version and the IDs of the retrieved sources for that response. Without those links, you cannot tell which model or prompt caused a bad rating. Store feedback events in a durable store such as DynamoDB or S3, not only in browser storage, and join them to invocation logs when you need the full context.
Annotation and rubrics
Reviewers label responses against a written rubric, with a scale and a clear example for each score. Check consistency by having two reviewers label the same sample and measure how often they agree. Where they disagree, make the rubric more specific. Bedrock human evaluation jobs use your own work team and the same rubric.
Reporting to stakeholders
Send feedback metrics to stakeholders in Amazon Quick Sight: satisfaction rate by model, by prompt version and by week, plus the top reasons for negative ratings. Automate a weekly report that compares models on the same metrics, so decisions use the same numbers. Keep private comments off shared dashboards, or restrict access with row-level security.
Turn feedback into tests
Turn each confirmed bad response into a regression test case: the input, the expected behavior and the sources it should use. Add it to the golden dataset so future prompt or model changes must pass it. Review new negative feedback each week, and merge duplicates so one common complaint does not dominate the suite.
How to choose
Scenario wording
Choose
Find which prompt version draws complaints
Feedback keyed to request ID and prompt version
Reviewers disagree on scores
A tighter rubric with examples, and an agreement check
Leaders want a satisfaction trend
Amazon Quick Sight dashboard
Stop a failure from coming back
Convert it to a regression test case
Compare two models every week
Automated weekly report on the same metrics
Dashboard shows private comments to everyone
Row-level security on the dataset
Common traps
Counting thumbs down without the request ID, so the cause stays unknown.
Treating every thumbs down as a bug. Some reflect out-of-scope questions or user error.
Letting one reviewer set the standard without calibration.
Keeping feedback only in browser storage, where it can be lost.
Dashboards that expose personal data to everyone who can open them.
A utility company's support team uses an AI tool that drafts ticket summaries for agents. Each draft can be rated with a thumbs up or a thumbs down, and the product team wants to learn which prompt versions and which knowledge documents produce poor summaries.
Each rating must be linked to the exact response it rates, including the model, the prompt version and the IDs of the documents that were retrieved. The product team needs the feedback available for a daily report within one day, and the team wants the least custom work for the logging.
Which design meets these requirements?
Choose one answer.
AA. Send each thumbs up or thumbs down to a CloudWatch metric that counts ratings per day, with no response identifier, so the dashboard shows the daily trend.
BB. Have each agent email their rating to the product team's shared mailbox, and let an analyst copy the ratings into a spreadsheet once a week.
CC. Return a request ID with each summary, and store each rating with that ID, the model, the prompt version and the retrieved document IDs in a DynamoDB table.
DD. Turn on model invocation logging, and rely on the logs to show which summaries agents disliked, since the logs record each response in full.
Show the answer and explanation
Answer: C
A. Wrong: A daily count shows the trend but cannot be tied to a response, a prompt version or a document. The product team would not learn which content is causing poor summaries.
B. Wrong: Weekly manual copying misses the one-day reporting requirement, and email does not carry the response details. The approach also depends on people, which makes errors likely.
C. Right: The request ID ties each rating to the exact response and its inputs, and the table is updated as ratings arrive. A daily report query meets the one-day requirement with only a small amount of logging code.
D. Wrong: Invocation logs record requests and responses but not the agents' ratings. Without the ratings, the logs cannot show which summaries were poor.
Explanation
Returning a request ID with each response lets the rating be written with the model, prompt version and document IDs that produced it. A small table and a daily query meet the reporting window with little custom code. Daily counts, weekly email copies and invocation logs each lose the link between the rating and the response.
Want more on User feedback, annotation and reporting? The 5 full-length practice sets have 5 more User feedback, annotation and reporting questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →