MLA-C02 practice question: Model and data drift monitoring (2 of 3)
Operating, Monitoring, and Securing ML and AI Solutions A free question from our original MLA-C02 practice exam. Try it, then open the answer to see why each option is right or wrong.
A pet insurer serves a claims-approval model from a real-time endpoint, and it already runs SageMaker Model Monitor for its other models. The true outcome of each claim, approved or denied after appeal, is recorded about 30 days after the decision, and the team must track accuracy against those outcomes each week.
Each later outcome must be matched to the prediction it belongs to, and the monitor must compute quality metrics such as accuracy and precision from the matched results.
Which configuration meets these requirements?
Choose one answer.
AA. Enable data capture, send an inference ID with each request, and upload outcomes tagged with the same IDs for a model quality monitor to merge.
BB. Compare the feature values in the captured requests with the training data statistics each week, since model quality follows from the quality of its input data.
CC. Send the outcomes to a new SageMaker AI endpoint, and treat that endpoint's predictions as the ground truth for the claims.
DD. Compute accuracy in a notebook each week from the predictions copied out of the endpoint logs, without loading the claim outcomes.
Show the answer and explanation
Answer: A
A. Right: Correct. Model quality monitoring merges captured predictions with ground truth labels stored in S3, and when a caller-supplied inference ID is present it uses that ID to match each label to its prediction. The monitor then computes metrics such as accuracy and precision. Learn more: Ingest Ground Truth labels and merge them with predictions (opens in a new tab)
B. Wrong: Data quality checks compare the inputs with the baseline and cannot compute accuracy, because they do not use the outcomes. The stem asks for outcome-based metrics.
C. Wrong: A second model's predictions are not the actual outcomes, so the metrics would measure agreement with another model. The stem requires the real outcomes.
D. Wrong: Without the outcomes the accuracy cannot be computed. The option also relies on manual steps the stem does not want.
Explanation
Model quality monitoring merges ground truth labels with captured predictions, matching them by the inference ID the caller supplied, and computes metrics such as accuracy from the matched results. Data quality checks, a second model's predictions or logs without outcomes cannot measure real accuracy.
Want more on Model and data drift monitoring? The 5 full-length practice sets have 11 more Model and data drift monitoring questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →