MLA-C02 practice question: Endpoint scaling and provisioning (2 of 2)
Deployment and Orchestration of ML and AI Workflows A free question from our original MLA-C02 practice exam. Try it, then open the answer to see why each option is right or wrong.
A ski resort technology group serves a large language model from a real-time SageMaker AI endpoint on GPU instances. Requests vary widely in length, each model container works on only a few requests at a time, and extra requests wait in a queue inside the container.
The current target tracking policy on invocations per instance reacts slowly: per-minute invocation counts look normal while queued requests pile up and latency climbs. The team wants scaling that follows how many requests each container is handling or holding, and that reacts within seconds.
Which change best meets these requirements?
Choose one answer.
AA. Replace the policy with target tracking on the high-resolution concurrent requests per model metric, with a target taken from load tests.
BB. Keep the invocations-per-instance policy, and shorten its scale-out cooldown so that it adds instances more often for the same traffic.
CC. Add a step scaling policy on the endpoint's CPUUtilization metric, since the CPU time of each request shows how long the GPU stays busy.
DD. Set the minimum and maximum instance counts to the peak value measured last season, so that the endpoint never needs to scale during traffic spikes.
Show the answer and explanation
Answer: A
A. Right: Correct. The concurrent requests per model metric counts the requests each model container is handling, including those queued inside it, and the high-resolution version is emitted every 10 seconds. Scaling on it adds instances as soon as the containers back up, whatever the request length. Learn more: Define a scaling policy (opens in a new tab)
B. Wrong: A shorter cooldown lets the policy act more often, but it still counts per-minute invocations, which look normal while long requests queue up. It does not track the load the stem describes. Learn more: Auto scaling policy overview (opens in a new tab)
C. Wrong: CPU utilization says little about the GPU work of a large language model, and it does not count the requests waiting in each container. The policy would miss the queue buildup that drives the latency.
D. Wrong: A fixed peak size pays for idle GPUs outside the busiest hours and still cannot react if traffic exceeds last season's peak. The team asked for scaling that follows the live load.
Explanation
When requests vary in length, invocation counts hide the real load. The high-resolution concurrent requests per model metric counts in-flight and queued requests in each container every 10 seconds, so target tracking on it scales out quickly as the containers back up. Shorter cooldowns, CPU-based scaling and fixed sizes do not follow this load.
Want more on Endpoint scaling and provisioning? The 5 full-length practice sets have 7 more Endpoint scaling and provisioning questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →