Endpoint scaling and provisioning: MLA-C02 study notes and practice questions
What to know about Endpoint scaling and provisioning for the MLA-C02 exam, from the Deployment and Orchestration of ML and AI Workflows domain (24% of the exam guide), then 2 free questions to test it. Part of the MLA-C02 study guide.
Scaling questions describe traffic patterns and latency problems and ask which policy or metric to use.
Policies
Policy
Use
Target tracking
Keep a metric near a target, such as invocations per instance; the usual choice
Step scaling
Add more capacity the further a metric breaches an alarm
Scheduled scaling
Raise minimum capacity before a known event (a sale, a game, the start of the work day)
Metrics
SageMakerVariantInvocationsPerInstance: requests per instance per minute; good when requests are similar in cost.
Concurrent requests per model (high-resolution, every 10 seconds): counts in-flight and queued requests, better when request length varies a lot, such as LLM generation. Scales faster.
CPU or GPU utilization: may not reflect model load accurately.
Async endpoints: scale on backlog size per instance, and can scale to zero.
Settings
Minimum and maximum instance counts; a minimum of at least two across AZs for availability.
Cooldown periods stop the policy from adding or removing too fast.
Use load tests (Inference Recommender) to find the target value per instance.
Provisioning and capacity
Inference components scale models independently on shared instances, including to zero copies.
Serverless endpoints scale automatically; provisioned concurrency keeps some warm.
SageMaker training plans reserve scarce GPU capacity ahead of time for planned work.
Common traps
Scaling on CPU when GPU-bound models show low CPU.
Reactive scaling alone for a peak you know is coming at 8 p.m.
Recreating endpoints to change capacity. Update the scaling policy or instance count.
Scaling in too quickly after a burst, so the next burst hits cold capacity; set a longer scale-in cooldown.
An esports league runs a match-prediction endpoint on a real-time SageMaker AI endpoint. Traffic jumps sharply during tournament weekends, and in the last tournament the endpoint slowed down for about 15 minutes before its instance count caught up.
The league wants the endpoint to add capacity before each weekend starts, and to keep scaling on the actual request load during the event. The scaling policy must track the request volume each instance handles.
Which TWO scaling elements should the team configure? (Choose two.)
Choose two answers.
AA. A target tracking policy on the SageMakerVariantInvocationsPerInstance metric, so that the instance count follows the number of requests each instance handles.
BB. A target tracking policy on the CPU utilization of the instances, since CPU always reflects the load of a model endpoint.
CC. A scheduled scaling action that raises the minimum instance count before each tournament weekend starts.
DD. Delete the endpoint each Friday and recreate it on Monday, so that every tournament starts with a fresh set of instances.
EE. Increase the size of each instance to the largest type available, so that no scaling is needed during the tournament.
Show the answer and explanation
Answers: A, C
A. Right: Correct. Target tracking on invocations per instance adds instances when each one handles more requests than the target. It keeps the endpoint near the chosen load during the event. Learn more: Define a scaling policy (opens in a new tab)
B. Wrong: CPU may not reflect the load of model inference, for example when the work runs on GPUs. The invocations-per-instance metric matches the request load the stem describes.
C. Right: Correct. A scheduled action raises capacity at a known time, one time or on a recurring schedule, so the endpoint is ready before traffic arrives. After it runs, the target tracking policy keeps adjusting capacity during the event. Learn more: Auto scaling policy overview (opens in a new tab)
D. Wrong: Recreating the endpoint each week takes time and risks downtime, and it does not scale on load during the event. The stem asks for scaling that follows the traffic.
E. Wrong: Oversized instances cost more all the time and still cannot match a traffic spike beyond their capacity. The stem asks for scaling by instance count.
Explanation
Target tracking on invocations per instance scales the endpoint on the request load each instance handles, and a scheduled action raises minimum capacity before a known event. CPU utilization may not reflect model load, and recreating or oversizing the endpoint does not scale on demand.
A ski resort technology group serves a large language model from a real-time SageMaker AI endpoint on GPU instances. Requests vary widely in length, each model container works on only a few requests at a time, and extra requests wait in a queue inside the container.
The current target tracking policy on invocations per instance reacts slowly: per-minute invocation counts look normal while queued requests pile up and latency climbs. The team wants scaling that follows how many requests each container is handling or holding, and that reacts within seconds.
Which change best meets these requirements?
Choose one answer.
AA. Replace the policy with target tracking on the high-resolution concurrent requests per model metric, with a target taken from load tests.
BB. Keep the invocations-per-instance policy, and shorten its scale-out cooldown so that it adds instances more often for the same traffic.
CC. Add a step scaling policy on the endpoint's CPUUtilization metric, since the CPU time of each request shows how long the GPU stays busy.
DD. Set the minimum and maximum instance counts to the peak value measured last season, so that the endpoint never needs to scale during traffic spikes.
Show the answer and explanation
Answer: A
A. Right: Correct. The concurrent requests per model metric counts the requests each model container is handling, including those queued inside it, and the high-resolution version is emitted every 10 seconds. Scaling on it adds instances as soon as the containers back up, whatever the request length. Learn more: Define a scaling policy (opens in a new tab)
B. Wrong: A shorter cooldown lets the policy act more often, but it still counts per-minute invocations, which look normal while long requests queue up. It does not track the load the stem describes. Learn more: Auto scaling policy overview (opens in a new tab)
C. Wrong: CPU utilization says little about the GPU work of a large language model, and it does not count the requests waiting in each container. The policy would miss the queue buildup that drives the latency.
D. Wrong: A fixed peak size pays for idle GPUs outside the busiest hours and still cannot react if traffic exceeds last season's peak. The team asked for scaling that follows the live load.
Explanation
When requests vary in length, invocation counts hide the real load. The high-resolution concurrent requests per model metric counts in-flight and queued requests in each container every 10 seconds, so target tracking on it scales out quickly as the containers back up. Shorter cooldowns, CPU-based scaling and fixed sizes do not follow this load.
Want more on Endpoint scaling and provisioning? The 5 full-length practice sets have 7 more Endpoint scaling and provisioning questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →