AIP-C01 practice question: User feedback, annotation and reporting
Testing, Validation, and Troubleshooting A free question from our original AIP-C01 practice exam. Try it, then open the answer to see why each option is right or wrong.
A utility company's support team uses an AI tool that drafts ticket summaries for agents. Each draft can be rated with a thumbs up or a thumbs down, and the product team wants to learn which prompt versions and which knowledge documents produce poor summaries.
Each rating must be linked to the exact response it rates, including the model, the prompt version and the IDs of the documents that were retrieved. The product team needs the feedback available for a daily report within one day, and the team wants the least custom work for the logging.
Which design meets these requirements?
Choose one answer.
AA. Send each thumbs up or thumbs down to a CloudWatch metric that counts ratings per day, with no response identifier, so the dashboard shows the daily trend.
BB. Have each agent email their rating to the product team's shared mailbox, and let an analyst copy the ratings into a spreadsheet once a week.
CC. Return a request ID with each summary, and store each rating with that ID, the model, the prompt version and the retrieved document IDs in a DynamoDB table.
DD. Turn on model invocation logging, and rely on the logs to show which summaries agents disliked, since the logs record each response in full.
Show the answer and explanation
Answer: C
A. Wrong: A daily count shows the trend but cannot be tied to a response, a prompt version or a document. The product team would not learn which content is causing poor summaries.
B. Wrong: Weekly manual copying misses the one-day reporting requirement, and email does not carry the response details. The approach also depends on people, which makes errors likely.
C. Right: The request ID ties each rating to the exact response and its inputs, and the table is updated as ratings arrive. A daily report query meets the one-day requirement with only a small amount of logging code.
D. Wrong: Invocation logs record requests and responses but not the agents' ratings. Without the ratings, the logs cannot show which summaries were poor.
Explanation
Returning a request ID with each response lets the rating be written with the model, prompt version and document IDs that produced it. A small table and a daily query meet the reporting window with little custom code. Daily counts, weekly email copies and invocation logs each lose the link between the rating and the response.
Want more on User feedback, annotation and reporting? The 5 full-length practice sets have 5 more User feedback, annotation and reporting questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →