Monitoring GenAI applications (CloudWatch, invocation logs, dashboards): AIP-C01 study notes and practice questions
What to know about Monitoring GenAI applications (CloudWatch, invocation logs, dashboards) for the AIP-C01 exam, from the Operational Efficiency and Optimization for GenAI Applications domain (12% of the exam guide), then 1 free question to test it. Part of the AIP-C01 study guide.
The exam expects you to watch three layers: the model, the application and the business outcome. For each question, pick the signal that answers it.
Model-layer metrics and logs
Bedrock publishes CloudWatch metrics for invocations, latency, throttles and input and output token counts per model. Use them for capacity and cost questions. Model invocation logging is off by default. When you turn it on, Bedrock writes request and response records to CloudWatch Logs or S3, which is how you analyze prompts and token use. CloudTrail records who called the API, not the prompt content.
Queries, anomalies and alarms
Use CloudWatch Logs Insights on invocation logs to find the prompts with the most tokens, error codes by model, and latency percentiles by application. Turn log fields into metrics with metric filters or the embedded metric format, so alarms can use them. Use CloudWatch anomaly detection on token metrics to catch sudden bursts, which often mean a loop or abuse. Use AWS Cost Anomaly Detection to alert on unexpected Bedrock spend.
Dashboards and business KPIs
CloudWatch dashboards show model and infrastructure metrics together. Business dashboards, such as deflection rate or average resolution time, belong in Amazon Quick Sight, fed from logs or a business data store. Tag requests with application inference profiles so each dashboard can split cost by application.
Tracing and synthetic checks
For requests that cross several services, use traces from X-Ray or OpenTelemetry (AWS Distro for OpenTelemetry) with CloudWatch Application Signals. CloudWatch Synthetics canaries run a user path on a schedule, such as asking a question and checking for an answer, and alarm when the path breaks.
How to choose
Scenario wording
Choose
Find which prompts consume the most tokens
Logs Insights on invocation logs
Alert when token use suddenly jumps
Anomaly detection alarm on token metrics
Alert on unexpected monthly spend
AWS Cost Anomaly Detection
Show deflection rate to leaders
Amazon Quick Sight dashboard on business data
Confirm the chat works every few minutes
CloudWatch Synthetics canary
Trace a slow request across services
X-Ray or ADOT traces with Application Signals
Common traps
Assuming invocation logs are on. They are off by default.
Using CloudTrail to read prompts. It records API calls, not content.
Logging full prompts with personal data without masking or retention limits.
Watching only latency, and missing quality or deflection drops.
Using the X-Ray SDKs for new code. They are in maintenance, so use OpenTelemetry.
A travel company runs a booking assistant on Amazon Bedrock. Several features call different models, and the operations team needs to see the cost and speed of each model as the day goes on.
The team wants a dashboard that shows, for each model, how many calls are made, how long they take, and how many input and output tokens they use, updated within a few minutes. Alarms should fire when latency rises or token use spikes.
The team does not want to parse full prompt logs to get these numbers, and the dashboard must cover every application that calls Bedrock in the account.
Which approach meets these requirements with the LEAST custom code?
Choose one answer.
AA. Schedule a nightly Athena query over the model invocation logs in Amazon S3, load the results into a spreadsheet each morning, and build the dashboard from that spreadsheet.
BB. Use AWS CloudTrail to count the Bedrock API call events for each model, and create CloudWatch alarms on those event counts to track how many calls are made.
CC. Use the Amazon Bedrock runtime metrics in CloudWatch, which report invocations, latency and input and output token counts per model, and build a CloudWatch dashboard with alarms on those metrics.
DD. Estimate the token counts in each application by dividing the character count by four, publish those estimates as custom metrics, and add a Lambda function that calculates average latency from the application logs.
Show the answer and explanation
Answer: C
A. Wrong: A nightly query gives yesterday's numbers, so the dashboard cannot update within minutes or trigger timely alarms. Parsing full logs also adds query cost and a process that someone must maintain. Learn more: What is Amazon Athena? (opens in a new tab)
D. Wrong: Character-based estimates are rough and differ from the tokens the model bills, and the Lambda function adds code to maintain in every application. The option is more work and less accurate than the managed metrics.
Explanation
The Bedrock runtime metrics in CloudWatch report invocations, latency and input and output token counts per model, so a CloudWatch dashboard and alarms can cover every application with no custom code. Nightly log queries are too slow, CloudTrail does not report latency or tokens, and character-based estimates are inaccurate.
Want more on Monitoring GenAI applications (CloudWatch, invocation logs, dashboards)? The 5 full-length practice sets have 5 more Monitoring GenAI applications (CloudWatch, invocation logs, dashboards) questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →
Study Monitoring GenAI applications (CloudWatch, invocation logs, dashboards)