Multi-model and multi-container endpoints: MLA-C02 study notes and practice questions
What to know about Multi-model and multi-container endpoints for the MLA-C02 exam, from the Deployment and Orchestration of ML and AI Workflows domain (24% of the exam guide), then 1 free question to test it. Part of the MLA-C02 study guide.
These questions describe dozens or thousands of models (one per customer, store or region) and ask how to host them without one endpoint each.
Options
Option
How it works
Fits
Multi-model endpoint (MME)
One container type serves many models stored in S3; a model loads into memory on its first request and is unloaded when memory is needed
Many similar, small, infrequently used models with the same framework; a cold load is acceptable
Multi-container endpoint
Up to 15 different containers on one endpoint, invoked directly or as a serial pipeline
Different frameworks that each get some traffic
Inference components
Deploy several models onto shared instances, each with its own CPU, GPU and memory allocation and its own scaling policy
Foundation models and larger models that need predictable resources; scale to zero per model
Inference pipeline
Containers run in sequence (preprocess, predict, postprocess)
One request flows through steps
Multi-model endpoint details
The request names the target model (TargetModel parameter).
Add a model by copying its artifact to the S3 prefix; no redeploy.
GPU MMEs are supported for some frameworks with Triton.
Watch ModelCacheHit and model loading metrics; frequent misses mean too little memory or too many active models.
Cost logic
Separate endpoints for 500 rarely used models means 500 sets of idle instances. An MME shares a few instances across all of them. If every model gets heavy, steady traffic, separate endpoints or inference components may give more predictable latency.
Common traps
One endpoint per model for hundreds of small models.
MME for models in different frameworks. Use multi-container endpoints or inference components.
Combining all models into one model, which changes their behavior.
Expecting no cold start on an MME for a model that hasn’t been called recently.
A dental chain has trained a separate small demand-forecasting model for each of its 300 clinics. Each model is only a few megabytes, and most clinics receive a prediction request only a few times a day.
The team wants all the models served from one SageMaker AI endpoint to keep hosting costs down, and it accepts that the first request for a rarely used clinic may load its model more slowly.
Which hosting approach meets these requirements?
Choose one answer.
AA. Deploy one multi-model endpoint, store the 300 model artifacts under one S3 prefix, and specify the model to invoke in each request.
BB. Deploy 300 separate real-time endpoints, one for each clinic model, so that each clinic has its own capacity.
CC. Combine all 300 models into one large model file that is loaded into memory when the endpoint starts, and route requests inside the model code.
DD. Run a batch transform job every hour over all 300 clinics, and let the front-end read the results from S3.
Show the answer and explanation
Answer: A
A. Right: Correct. A multi-model endpoint hosts many models behind one endpoint and loads each model from S3 when it is first requested. This suits many small, infrequently used models and accepts a slower first request. Learn more: Multi-model endpoints (opens in a new tab)
B. Wrong: Three hundred endpoints multiply the hosting cost and management effort for models that receive few requests. The stem asks for one endpoint to keep costs down.
C. Wrong: A combined model changes how the forecasts are produced and mixes unrelated clinics, which the team does not want. The option also loads everything at start-up rather than on demand.
D. Wrong: Batch transform is offline and not built to serve a request for one clinic at a time. The stem describes on-demand prediction requests.
Explanation
A multi-model endpoint hosts many models behind one endpoint, loading each model from S3 on first use. It suits many small, infrequently used models. Separate endpoints multiply cost, a combined model changes the forecasts, and batch transform is not an on-demand service.
Want more on Multi-model and multi-container endpoints? The 5 full-length practice sets have 5 more Multi-model and multi-container endpoints questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →