Evaluating agents (task completion, tool use, AgentCore Evaluations): AIP-C01 study notes and practice questions
What to know about Evaluating agents (task completion, tool use, AgentCore Evaluations) for the AIP-C01 exam, from the Testing, Validation, and Troubleshooting domain (11% of the exam guide), then 1 free question to test it. Part of the AIP-C01 study guide.
An agent needs more than an answer check. Grade the outcome, each tool choice and the steps between them, and test before release and after changes.
Scripted scenarios
Batch evaluation runs an agent through scripted scenarios that have defined goals and expected end states. Score task completion as pass or fail, and count steps and tool calls against a limit. Run the batch before each release and whenever the base model changes, because a new model can change tool choices with no code change.
Tool use and reasoning
Tool selection accuracy checks that the agent called the right tool with valid arguments. Reasoning quality checks that a multi-step plan made sense, such as stopping when the task is done instead of looping. AgentCore Evaluations provides built-in evaluators for these measures, and you can write custom evaluators for your own success rules.
Online evaluation
Online evaluation samples a share of live sessions and scores them continuously. It catches drift that scripted tests miss. Sampling keeps cost down. Use it to watch quality after release, and alert when the score falls below a threshold.
A/B tests of agent versions
To choose between two agent versions, send a share of live traffic to each, keep the tools and data the same, and compare completion rate, tool errors, latency and cost. Route each user to one version consistently, and collect enough sessions for a fair comparison before you declare a winner.
How to choose
Scenario wording
Choose
Check the agent before release on fixed scenarios
Batch evaluation with scripted scenarios
Watch the quality of live sessions after release
Online evaluation with sampling
The agent sometimes calls the wrong tool
Tool selection accuracy metric
The agent loops or wanders across steps
Reasoning quality and step count
Two agent versions, and you need the better one
A/B test on live traffic
Success depends on a business rule
Custom evaluator
Common traps
Scoring only the final text, which hides wrong tool calls that happened to succeed.
Testing once and not again after a base model change.
Too few sessions to see a difference, followed by a conclusion anyway.
Moving a user between versions mid-task, which mixes the results.
Choosing Amazon Bedrock Agents Classic for a new agent. It is closed to new customers, so build on AgentCore.
A payroll software company runs an agent that answers HR administrators' questions and calls tools to read and update employee records. The agent is deployed on Amazon Bedrock AgentCore, and its traces are available in AgentCore Observability.
The team wants to know whether the agent completes tasks correctly and uses the right tools on real customer traffic, and it wants to catch quality drops after each prompt change. The evaluation must not slow down the agent's replies, and it should score only a sample of sessions to keep the cost low.
Which approach meets these requirements?
Choose one answer.
AA. Run an offline evaluation of 1,000 scripted sessions before each release, and approve the release only when the offline scores meet the threshold.
BB. Configure an online evaluation that samples a share of live sessions, scores them with built-in evaluators, and shows the results beside the traces.
CC. Create a CloudWatch alarm on the agent's Lambda error rate and page the on-call engineer when it rises, since errors usually signal a poor agent response.
DD. Export all session transcripts to Amazon S3 each night, and have analysts read a random sample of them each week to score task completion by hand.
Show the answer and explanation
Answer: B
A. Wrong: Scripted sessions can catch some regressions, but they do not show how the agent performs on live customer traffic. The team wants continuous evaluation of real sessions.
B. Right: Online evaluation runs beside live traffic, so replies are not delayed, and sampling keeps the scoring cost down. The results appear with the traces, which shows drops after each prompt change. Learn more: Create online evaluation (opens in a new tab)
C. Wrong: Error rate counts failed invocations, but an agent can complete a call and still pick the wrong tool or the wrong record. The alarm does not measure task completion at all.
D. Wrong: The sample is taken once a week, so regressions after a prompt change can go unseen for days. Manual scoring also does not scale with traffic and does not use the agent's trace data.
Explanation
Online evaluation scores a sample of live agent sessions in the background, using the traces the agent already produces. This catches task completion and tool use problems on real traffic without slowing replies or adding much cost. Offline tests, error alarms and weekly manual reads each leave regressions unseen for too long or miss the quality question.
Want more on Evaluating agents (task completion, tool use, AgentCore Evaluations)? The 5 full-length practice sets have 5 more Evaluating agents (task completion, tool use, AgentCore Evaluations) questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →
Study Evaluating agents (task completion, tool use, AgentCore Evaluations)