Exams / SAP-C03 / Study guide / Monitoring AI workloads and user experience (Synthetics, RUM)
Monitoring AI workloads and user experience (Synthetics, RUM): SAP-C03 study notes and practice questions
What to know about Monitoring AI workloads and user experience (Synthetics, RUM) for the SAP-C03 exam, from the Operational Excellence and Automation domain (20% of the exam guide), then 2 free questions to test it. Part of the SAP-C03 study guide.
SAP-C03 asks what to measure for generative AI and ML workloads (latency, tokens, quality, drift, agent behavior) and how to know what users actually experience, not just what servers report.
Generative AI metrics
Amazon Bedrock publishes CloudWatch metrics per model, including invocations, invocation latency, input and output token counts, client and server errors and throttles.
Model invocation logging sends prompts and responses to CloudWatch Logs or S3 for audit, quality review and cost analysis; protect these logs because they may contain sensitive data.
Guardrail interventions show how often content is blocked.
Watch time to first token and total latency for streaming chat, and throttles as a capacity signal (for example, to adopt cross-Region inference).
Agents
Amazon Bedrock AgentCore Observability traces each agent session: model calls, tool calls, memory access, latency and errors, in OpenTelemetry format and shown in CloudWatch.
Track agent-level measures such as task success, steps per task, tool error rates and cost per session. Quality scoring before release belongs in pipeline evaluation.
ML inference and drift
SageMaker AI endpoints publish invocations, model latency, overhead latency and 4xx and 5xx errors.
SageMaker Model Monitor compares live traffic with a baseline and flags data quality drift, model quality drift (needs ground-truth labels), bias drift and feature attribution drift, with CloudWatch alarms that can trigger retraining.
User experience
CloudWatch Synthetics
CloudWatch RUM
Traffic
Scripted canaries on a schedule, as often as once a minute
Real users’ browsers
Finds
Outages and broken flows before users report them, even with no traffic
Page load times, Core Web Vitals, JavaScript and HTTP errors, by browser, device and location
Built on
Lambda functions running Node.js, Python or Java scripts (headless browser or API calls)
A JavaScript snippet in the web app
Canaries can check APIs, multi-step user journeys, broken links and visual changes, and store screenshots.
Both feed Application Signals, so a slow page can be followed to the backend service and trace.
How to choose
Scenario wording
Choose
“Alert on rising token use or throttling for a Bedrock app”
Bedrock CloudWatch metrics with alarms
“Keep a record of every prompt and response”
Bedrock model invocation logging
“See which tool call slowed an agent down”
AgentCore Observability traces
“Detect when production input data no longer looks like training data”
SageMaker Model Monitor data quality
“Know the checkout flow is broken before customers do”
Synthetics canary
“Page load is slow for users in one country”
CloudWatch RUM
Common traps
Server CPU metrics as the measure of AI response quality or user experience.
RUM to detect outages at 3 a.m. when there are no users; that’s what canaries are for.
Model quality monitoring without ground truth labels.
Logging prompts with no encryption or access controls.
Custom tracing code in agents when AgentCore Observability provides it.
A logistics firm runs a customer-support agent on Amazon Bedrock AgentCore. The agent calls tracking and claims tools, and it is released in several versions a month. Support managers report that some customers get slow answers and that tool calls sometimes fail without an error reaching the dashboard.
The platform team wants per-version visibility into response latency, token use and tool-call failures, with alarms that fire when answer quality drifts below an agreed level. It wants managed observability rather than a metrics pipeline it builds itself.
Which approach meets these requirements?
Choose one answer.
AA. Read the agent's chat transcripts once a month, count the complaints that mention slow answers or failed tools for each agent version, and share the counts with the support managers in a report.
BB. Turn on AgentCore Observability so latency, token use and tool-call errors appear in CloudWatch for each agent version, and run scheduled evaluations on a fixed question set that publish a quality metric with an alarm.
CC. Add a print statement to each tool function that writes its duration to the application log, and search the log for the word error once a day.
DD. Create a CloudWatch Synthetics canary that asks the agent one fixed question every hour, and alarm when the canary gets no answer.
Show the answer and explanation
Answer: B
A. Wrong: Monthly transcript reviews are slow and do not give per-version metrics or alarms. The option does not measure quality or latency as the agent runs.
B. Right: Correct. AgentCore Observability sends the agent's operational metrics and traces, including latency, token use and errors, to CloudWatch without a custom pipeline. Scheduled evaluations turn answer quality into a metric, so an alarm can fire when quality drifts below the agreed level.
C. Wrong: Print statements are a home-made measurement that needs code in every tool, and a daily search for the word error misses slow answers and quality drift. The option gives no per-version alarms.
D. Wrong: One fixed question shows only that the agent answers at all, not how fast or how well it answers across versions. Quality drift would stay hidden until a failure.
Explanation
AgentCore Observability provides latency, token and error metrics for the agent in CloudWatch, and scheduled quality evaluations published as metrics let alarms catch drift. Monthly reviews, daily error searches and a single canary question each miss the per-version or quality view.
A news site's mobile web app loads its article pages from Amazon CloudFront. Editors suspect that readers in some countries see slow pages or broken layouts, but the origin servers report healthy metrics, and the team cannot reproduce the problem from its office. Most readers reach the site on phones rather than desktop computers.
The team wants to measure page load times and JavaScript errors as readers actually experience them, grouped by country and browser, with no change to the reader's experience.
Which solution meets these requirements?
Choose one answer.
AA. Create an alarm on the load balancer's latency metric, and check it each morning for days when the average rises above its usual level.
BB. Run a CloudWatch Synthetics canary that loads the article page from the office network every five minutes in a headless browser, and alarm when the page load time exceeds the threshold.
CC. Add CloudWatch RUM to the site so the browser records page load times and JavaScript errors from real reader sessions, grouped by country and browser, with alarms on those metrics.
DD. Turn on X-Ray tracing for the origin servers, and read the trace for each slow article to see which server call takes the longest.
Show the answer and explanation
Answer: C
A. Wrong: The load balancer measures only the server side, so it cannot see browser rendering, layout breaks or errors in the reader's browser. The healthy origin metrics are the reason the team is stuck.
B. Wrong: A canary in the office runs from one location and one browser, so it cannot show what readers in other countries and browsers experience. The team has already failed to reproduce the problem from that location.
C. Right: Correct. RUM collects performance and error data from real reader sessions, and its dimensions include country and browser. The team gets the experience data without changing how readers use the site.
D. Wrong: Server traces explain server-side time only and do not show the browser's rendering or its JavaScript errors. The option repeats the blind spot the team already has.
Explanation
CloudWatch RUM records page load and JavaScript errors from real reader sessions, grouped by country and browser, which the origin metrics and a single-location canary cannot show. Load balancer latency and server traces stop at the server, and one office canary sees one location.
Want more on Monitoring AI workloads and user experience (Synthetics, RUM)? The 5 full-length practice sets have 6 more Monitoring AI workloads and user experience (Synthetics, RUM) questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →
Study Monitoring AI workloads and user experience (Synthetics, RUM)