Exams / AIP-C01 / Study guide / Continuous monitoring for misuse, drift and policy violations
Continuous monitoring for misuse, drift and policy violations: AIP-C01 study notes and practice questions
What to know about Continuous monitoring for misuse, drift and policy violations for the AIP-C01 exam, from the AI Safety, Security, and Governance domain (20% of the exam guide), then 1 free question to test it. Part of the AIP-C01 study guide.
AIP-C01 asks how to keep a generative system compliant after launch. A one-time review before release is not enough, because models, prompts, data and user behavior all change. The exam expects monitoring that detects a problem, alerts a person, and where safe, fixes it automatically.
Watching for misuse
Publish guardrail intervention metrics to CloudWatch, which record how often input or output was blocked or masked. Alarm on a sudden rise, using a static threshold for known limits or anomaly detection for normal variation. Track request volume per user or API key, and alarm when one identity sends far more requests than its peers, which often signals abuse or automated attacks.
Detecting quality and bias drift
Schedule evaluations on a fixed test set and on a sample of production traffic. Use Bedrock evaluations with LLM-as-a-judge metrics, and include fairness and harmfulness criteria. Compare each run with a baseline and alarm when a score drops. For drift in a classic ML model, SageMaker Model Monitor and Clarify are closed to new customers, so use Bedrock evaluations with CloudWatch metrics and alarms instead. See Bedrock evaluations.
Automated remediation
An alarm can trigger an EventBridge rule that starts a Step Functions workflow. Typical steps include switching traffic to a safer prompt version, raising a guardrail strength, suspending one user’s access or opening a review ticket. Use a human approval callback for high-impact actions, such as disabling a model for all users, so a false alarm does not cause a broad outage.
Redaction and retention in monitoring
Monitoring data is still data. Redact personal values before logs are stored, keep log groups restricted, and set retention so monitoring records expire on schedule.
Keep a baseline
Version prompts, guardrails and evaluation datasets. A drop in score is only meaningful against a known baseline, and a version history lets you roll back the change that caused it.
How to choose
Scenario wording
Choose
“Alert when guardrail blocks jump sharply”
CloudWatch alarm on guardrail intervention metrics
“Answer quality is slowly getting worse”
Scheduled Bedrock evaluation compared with a baseline
“Respond automatically to a policy violation”
EventBridge rule starting a Step Functions remediation
“One user sends many attack prompts”
Per-user usage anomaly detection and alarm
“Detect drift in a production classifier”
Bedrock evaluations with CloudWatch alarms, since Model Monitor is closed
Common traps
Recommending SageMaker Model Monitor or Clarify for new builds. Both are closed to new customers.
Relying on a one-time evaluation before launch.
Automatic full shutdown on a single block. One event is not a trend, so alarm on patterns and require approval for broad actions.
Logging unredacted content to get better monitoring data.
Changing a prompt or guardrail with no version record, which makes the next drop impossible to explain.
A mobile game studio runs a chat assistant that helps players with in-game questions, using a model on Amazon Bedrock with a guardrail attached. Some players try to use the assistant to generate abusive messages or to extract cheat codes.
The trust and safety team wants to know within minutes when the number of blocked requests jumps, because a coordinated campaign often starts with a burst of attempts. The on-call engineer must be paged automatically when that happens.
The monitoring must use the guardrail's own metrics, with no extra code in the game backend and no need to read the logs by hand.
Which approach meets these requirements?
Choose one answer.
AA. Export the invocation logs to Amazon S3 each day, query them with Amazon Athena every morning, and email the trust and safety team a list of the players whose requests were blocked most often.
BB. Create a CloudWatch alarm on the guardrail's intervention metric, with a threshold or an anomaly detection band, and send the alarm action to an SNS topic that pages the on-call engineer.
CC. Raise every content filter to HIGH strength so that fewer attempts reach the model, and review the monthly blocked-request totals in the billing console once the month closes.
DD. Add a Lambda function to the game backend that scores each reply with a second model, writes a row to Amazon DynamoDB for every exchange, and build a dashboard from that table.
Show the answer and explanation
Answer: B
A. Wrong: Daily exports and morning queries detect a burst hours after it starts, which is too slow for a coordinated campaign. Emailing a list also leaves the paging requirement unmet. Learn more: What is Amazon Athena? (opens in a new tab)
D. Wrong: The function works, but it adds a second model call and a database write to every exchange, which costs more and changes the game backend. It also reproduces metrics that the guardrail already publishes.
Explanation
The guardrail publishes intervention metrics to CloudWatch, so an alarm on those metrics can detect a burst within minutes without touching the game backend. Routing the alarm to an SNS topic pages the on-call engineer automatically. Daily log queries, stronger filters and custom scoring are slower, change the system, or do not alert anyone.
Want more on Continuous monitoring for misuse, drift and policy violations? The 5 full-length practice sets have 6 more Continuous monitoring for misuse, drift and policy violations questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →
Study Continuous monitoring for misuse, drift and policy violations