AIP-C01 practice question: Monitoring GenAI applications (CloudWatch, invocation logs, dashboards)
Operational Efficiency and Optimization for GenAI Applications A free question from our original AIP-C01 practice exam. Try it, then open the answer to see why each option is right or wrong.
A travel company runs a booking assistant on Amazon Bedrock. Several features call different models, and the operations team needs to see the cost and speed of each model as the day goes on.
The team wants a dashboard that shows, for each model, how many calls are made, how long they take, and how many input and output tokens they use, updated within a few minutes. Alarms should fire when latency rises or token use spikes.
The team does not want to parse full prompt logs to get these numbers, and the dashboard must cover every application that calls Bedrock in the account.
Which approach meets these requirements with the LEAST custom code?
Choose one answer.
AA. Schedule a nightly Athena query over the model invocation logs in Amazon S3, load the results into a spreadsheet each morning, and build the dashboard from that spreadsheet.
BB. Use AWS CloudTrail to count the Bedrock API call events for each model, and create CloudWatch alarms on those event counts to track how many calls are made.
CC. Use the Amazon Bedrock runtime metrics in CloudWatch, which report invocations, latency and input and output token counts per model, and build a CloudWatch dashboard with alarms on those metrics.
DD. Estimate the token counts in each application by dividing the character count by four, publish those estimates as custom metrics, and add a Lambda function that calculates average latency from the application logs.
Show the answer and explanation
Answer: C
A. Wrong: A nightly query gives yesterday's numbers, so the dashboard cannot update within minutes or trigger timely alarms. Parsing full logs also adds query cost and a process that someone must maintain. Learn more: What is Amazon Athena? (opens in a new tab)
D. Wrong: Character-based estimates are rough and differ from the tokens the model bills, and the Lambda function adds code to maintain in every application. The option is more work and less accurate than the managed metrics.
Explanation
The Bedrock runtime metrics in CloudWatch report invocations, latency and input and output token counts per model, so a CloudWatch dashboard and alarms can cover every application with no custom code. Nightly log queries are too slow, CloudTrail does not report latency or tokens, and character-based estimates are inaccurate.
Want more on Monitoring GenAI applications (CloudWatch, invocation logs, dashboards)? The 5 full-length practice sets have 5 more Monitoring GenAI applications (CloudWatch, invocation logs, dashboards) questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →