Model and data drift monitoring: MLA-C02 study notes and practice questions
What to know about Model and data drift monitoring for the MLA-C02 exam, from the Operating, Monitoring, and Securing ML and AI Solutions domain (24% of the exam guide), then 3 free questions to test it. Part of the MLA-C02 study guide.
Monitoring questions describe a model whose results are slipping, and ask which monitor detects it or how to set it up.
How it works
Turn on data capture on the endpoint (or batch transform) to save requests and responses to S3.
Run a baseline job on the training data to compute statistics and constraints.
Create a monitoring schedule that compares captured data with the baseline.
Results and violations go to S3 and CloudWatch metrics; alarms notify the team or trigger retraining through EventBridge and SageMaker Pipelines.
The four monitors
Monitor
Detects
Needs
Data quality
Input data drift: missing values, type changes, distribution shifts
Captured inputs and a baseline from training data
Model quality
Falling accuracy, precision, recall, RMSE
Ground truth labels merged with predictions by inference ID
Bias drift
Changes in Clarify post-training bias metrics for a facet
Captured predictions and a bias config
Feature attribution drift
Changes in SHAP feature importance
Captured data and a Clarify baseline
Ground truth often arrives later (a loan defaults months afterward); upload it to S3 with the inference IDs the caller sent, and the model quality job matches them.
Kinds of drift
Data (covariate) drift: inputs change, such as new customer types.
Concept drift: the relationship between inputs and the target changes, such as fraud patterns.
Label drift: the target’s distribution changes.
Responding
Alert and investigate first; not every drift needs retraining.
Retrain on recent data through a pipeline triggered by the alarm.
Review features, thresholds and data sources.
Common traps
Data quality monitoring to measure accuracy. It can’t see labels; use model quality.
No inference IDs, so ground truth can’t be matched.
Writing custom drift scripts when Model Monitor provides them.
A logistics broker serves an ETA model from a real-time SageMaker AI endpoint, and its ML team already uses SageMaker Model Monitor on its other production endpoints. Last week a carrier changed how it reports dwell time, and missing values in one input feature rose from under 1 percent to about 20 percent without any endpoint error.
The team wants a daily automatic check that compares the endpoint's inference inputs with statistics and constraints from the training data, and an alert when they no longer match.
Which solution meets these requirements?
Choose one answer.
AA. Turn on endpoint logs in CloudWatch, and have an analyst search them each morning for the word "missing" in the feature payloads.
BB. Run a weekly SageMaker Processing job that retrains the model and compares the training accuracy with the last week's accuracy.
CC. Enable AWS Config rules on the endpoint to check that the model artifact has not changed since deployment.
DD. Enable data capture, and create a Model Monitor data quality schedule on the endpoint with a baseline from the training data and CloudWatch alarms on violations.
Show the answer and explanation
Answer: D
A. Wrong: Log text search depends on the payload containing that word, and it does not compare the data against training statistics. The option is manual and would miss subtle shifts.
B. Wrong: Retraining does not check the incoming data, and training accuracy does not show a change in input quality. The option does not detect the missing-value shift.
C. Wrong: AWS Config checks resource configuration, such as whether the artifact changed, and does not look at the data flowing into the model. The problem is in the inputs.
D. Right: Correct. Data quality monitoring compares captured inference data with a baseline of statistics and constraints from the training data, and it reports violations on a schedule. Alarms on the violation metrics notify the team. Learn more: Data quality monitoring with Model Monitor (opens in a new tab)
Explanation
Model Monitor data quality monitoring compares captured inference data with baseline statistics and constraints computed from the training data, and reports violations on a schedule. Alarms on those results tell the team when the inputs change. Log searches, retraining and configuration checks do not compare the data with the baseline.
A pet insurer serves a claims-approval model from a real-time endpoint, and it already runs SageMaker Model Monitor for its other models. The true outcome of each claim, approved or denied after appeal, is recorded about 30 days after the decision, and the team must track accuracy against those outcomes each week.
Each later outcome must be matched to the prediction it belongs to, and the monitor must compute quality metrics such as accuracy and precision from the matched results.
Which configuration meets these requirements?
Choose one answer.
AA. Enable data capture, send an inference ID with each request, and upload outcomes tagged with the same IDs for a model quality monitor to merge.
BB. Compare the feature values in the captured requests with the training data statistics each week, since model quality follows from the quality of its input data.
CC. Send the outcomes to a new SageMaker AI endpoint, and treat that endpoint's predictions as the ground truth for the claims.
DD. Compute accuracy in a notebook each week from the predictions copied out of the endpoint logs, without loading the claim outcomes.
Show the answer and explanation
Answer: A
A. Right: Correct. Model quality monitoring merges captured predictions with ground truth labels stored in S3, and when a caller-supplied inference ID is present it uses that ID to match each label to its prediction. The monitor then computes metrics such as accuracy and precision. Learn more: Ingest Ground Truth labels and merge them with predictions (opens in a new tab)
B. Wrong: Data quality checks compare the inputs with the baseline and cannot compute accuracy, because they do not use the outcomes. The stem asks for outcome-based metrics.
C. Wrong: A second model's predictions are not the actual outcomes, so the metrics would measure agreement with another model. The stem requires the real outcomes.
D. Wrong: Without the outcomes the accuracy cannot be computed. The option also relies on manual steps the stem does not want.
Explanation
Model quality monitoring merges ground truth labels with captured predictions, matching them by the inference ID the caller supplied, and computes metrics such as accuracy from the matched results. Data quality checks, a second model's predictions or logs without outcomes cannot measure real accuracy.
A regional bank monitors a credit-decision model in production with SageMaker Model Monitor and SageMaker Clarify, which it already uses. Its pre-deployment analysis found no bias on the age facet, but the model-risk team now wants to detect whether approval rates for each age group drift over the coming months.
The team wants a scheduled check that compares the model's predictions on production traffic with the bias baseline recorded before deployment, and an alert when the bias moves beyond a set threshold.
Which SageMaker AI capability should the team use?
Choose one answer.
AA. A pre-training bias analysis of the original training data, run again each month with the same Clarify settings, facet and label values as before.
BB. Model Monitor bias drift monitoring, using a Clarify configuration with the age facet and a baseline from the pre-deployment analysis.
CC. Feature attribution drift monitoring, which reports when the importance of each feature changes over time.
DD. Data quality monitoring on the endpoint, with the age column added to the baseline statistics.
Show the answer and explanation
Answer: B
A. Wrong: Pre-training analysis checks the dataset and does not look at the model's predictions on production traffic. Running it again repeats the wrong measurement.
B. Right: Correct. Bias drift monitoring runs on a schedule, compares the production predictions for the chosen facet with the baseline, and reports violations. This matches the drift check the risk team wants. Learn more: Bias drift for models in production (opens in a new tab)
C. Wrong: Attribution drift tracks changes in how features influence predictions, which is useful, but it does not measure the approval rates across age groups. The stem asks about outcomes by group.
D. Wrong: Data quality monitoring compares input distributions and constraints, and it does not measure the model's predictions by group. The bias question needs bias drift monitoring.
Explanation
Model Monitor bias drift monitoring runs Clarify bias metrics on production predictions on a schedule and compares them with a baseline for the chosen facet. Pre-training analysis repeats a dataset check, and attribution or data quality monitoring measure different things than group-level approval rates.
Want more on Model and data drift monitoring? The 5 full-length practice sets have 11 more Model and data drift monitoring questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →