Testing, Validation, and Troubleshooting A free question from our original AIP-C01 practice exam. Try it, then open the answer to see why each option is right or wrong.
A payroll software company runs an agent that answers HR administrators' questions and calls tools to read and update employee records. The agent is deployed on Amazon Bedrock AgentCore, and its traces are available in AgentCore Observability.
The team wants to know whether the agent completes tasks correctly and uses the right tools on real customer traffic, and it wants to catch quality drops after each prompt change. The evaluation must not slow down the agent's replies, and it should score only a sample of sessions to keep the cost low.
Which approach meets these requirements?
Choose one answer.
AA. Run an offline evaluation of 1,000 scripted sessions before each release, and approve the release only when the offline scores meet the threshold.
BB. Configure an online evaluation that samples a share of live sessions, scores them with built-in evaluators, and shows the results beside the traces.
CC. Create a CloudWatch alarm on the agent's Lambda error rate and page the on-call engineer when it rises, since errors usually signal a poor agent response.
DD. Export all session transcripts to Amazon S3 each night, and have analysts read a random sample of them each week to score task completion by hand.
Show the answer and explanation
Answer: B
A. Wrong: Scripted sessions can catch some regressions, but they do not show how the agent performs on live customer traffic. The team wants continuous evaluation of real sessions.
B. Right: Online evaluation runs beside live traffic, so replies are not delayed, and sampling keeps the scoring cost down. The results appear with the traces, which shows drops after each prompt change. Learn more: Create online evaluation (opens in a new tab)
C. Wrong: Error rate counts failed invocations, but an agent can complete a call and still pick the wrong tool or the wrong record. The alarm does not measure task completion at all.
D. Wrong: The sample is taken once a week, so regressions after a prompt change can go unseen for days. Manual scoring also does not scale with traffic and does not use the agent's trace data.
Explanation
Online evaluation scores a sample of live agent sessions in the background, using the traces the agent already produces. This catches task completion and tool use problems on real traffic without slowing replies or adding much cost. Offline tests, error alarms and weekly manual reads each leave regressions unseen for too long or miss the quality question.
Want more on Evaluating agents (task completion, tool use, AgentCore Evaluations)? The 5 full-length practice sets have 5 more Evaluating agents (task completion, tool use, AgentCore Evaluations) questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →