Monitoring FMs and agents: MLA-C02 study notes and practice questions
What to know about Monitoring FMs and agents for the MLA-C02 exam, from the Operating, Monitoring, and Securing ML and AI Solutions domain (24% of the exam guide), then 2 free questions to test it. Part of the MLA-C02 study guide.
These questions describe a chatbot or agent that is slow, expensive or misbehaving and ask what to monitor or how to find the cause.
Bedrock metrics in CloudWatch
Metric
Tells you
Invocations
Request volume
InvocationLatency
How long responses take
InputTokenCount / OutputTokenCount
Cost drivers; long outputs mean higher cost
InvocationThrottles
Hitting quotas; consider cross-Region inference or provisioned throughput
InvocationClientErrors / ServerErrors
Bad requests or service problems
Set alarms on latency, tokens and throttles, and build dashboards per application.
Logging
Model invocation logging (off by default) records prompts, responses and metadata to CloudWatch Logs or S3. Protect the destination, since logs may contain sensitive data.
Logs feed evaluation, debugging and distillation datasets.
Agents
Bedrock agent traces show each orchestration step, action group call, knowledge base lookup and observation, so you can see where a request went wrong.
AgentCore Observability gives traces, sessions and metrics in CloudWatch for agents on AgentCore Runtime, using OpenTelemetry.
CloudWatch generative AI observability dashboards bring model and agent data together.
Quality and safety
Guardrail interventions show how often content is blocked.
Run evaluation jobs on samples of real traffic to track answer quality.
Collect user feedback (thumbs up or down) and review low-rated answers.
Watch knowledge base retrieval results for missing or stale sources.
Cost control
Track tokens per user or tenant (application inference profiles allow cost allocation tags).
Set max_tokens, use prompt caching, and route simple prompts to smaller models.
Common traps
CPU or memory metrics for a Bedrock app. There are no instances to watch; watch tokens and latency.
Changing the model or temperature before reading the agent trace.
A public library runs a Bedrock-based assistant for patrons. Last month its monthly model bill rose sharply, and the operations team suspects that a few prompts are generating very long responses. Patrons also report that some answers arrive slowly during peak hours.
The team wants CloudWatch alarms that detect both the cost problem and the slowdown, using metrics that Amazon Bedrock publishes for the model invocations, without changing the application code.
Which TWO metrics should the team alarm on? (Choose two.)
Choose two answers.
AA. The OutputTokenCount metric for the model, since the number of output tokens drives the cost of long responses.
BB. The InvocationLatency metric for the model, which measures the time from each request to its last token, so the team sees the peak-hour slowdown.
CC. The CPUUtilization metric for the Bedrock service, which shows how hard the foundation model is working on each request.
DD. The number of records in the library's catalog table, which grows with usage and therefore shows the cost of the assistant.
EE. The Model Monitor data quality violations on the library's catalog, since bad catalog data causes long responses.
C. Wrong: Bedrock is a managed service, and the team has no CPUUtilization metric for foundation model work to alarm on. The option names a metric that is not the right signal.
D. Wrong: The catalog size is not a Bedrock model metric and does not show token use or latency. Catalog growth also does not follow the cost spike the stem describes.
E. Wrong: Model Monitor applies to SageMaker AI endpoints, and the assistant runs on Bedrock. The option does not track the token use or latency the team needs.
Explanation
Bedrock publishes input and output token counts and invocation latency in CloudWatch. Output tokens reveal the cost of long responses, and latency reveals the slowdown, so alarms on those two metrics cover both problems. CPU, catalog size and Model Monitor do not measure Bedrock invocations.
A ski resort has run an Amazon Bedrock agent for two years to answer guest questions about lift tickets, rentals and trail conditions. The agent calls action group functions for ticket lookup and rental booking, and guests report that some bookings never confirm even though the agent says they were made.
The operations team needs to see, for a failing conversation, each step the agent took, the action it called and the response it received, without changing the agent's instructions or the guest-facing code.
Which capability should the team use?
Choose one answer.
AA. Turn on the model's temperature setting to zero, since deterministic output makes the failing conversations easy to reproduce.
BB. Add a second foundation model that reviews every guest answer and writes a short summary of each conversation for the operations team.
CC. Enable the agent trace on invocations so that orchestration steps, action group calls and their observations are returned with each response and can be logged.
DD. Move the booking logic out of the agent into the guest-facing application, so the application logs show every booking.
Show the answer and explanation
Answer: C
A. Wrong: Temperature changes the randomness of answers and does not show which action failed or what it returned. The stem asks for visibility into the steps.
B. Wrong: A summary from another model describes the answer, not the action calls and responses that failed. The team needs the trace data itself.
D. Wrong: Moving the logic changes the architecture, which the stem rules out. Application logs would also not show the agent's reasoning steps.
Explanation
Agent traces return each orchestration step, action group call and observation, which shows where a booking request failed. Changing temperature, adding a summarizing model or moving the logic into the application either changes behavior or does not show the agent's steps.
Want more on Monitoring FMs and agents? The 5 full-length practice sets have 10 more Monitoring FMs and agents questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →