Exams / AIP-C01 / Study guide / Hosting LLMs on containers and GPUs (SageMaker AI, EKS, model loading)
Hosting LLMs on containers and GPUs (SageMaker AI, EKS, model loading): AIP-C01 study notes and practice questions
What to know about Hosting LLMs on containers and GPUs (SageMaker AI, EKS, model loading) for the AIP-C01 exam, from the Implementation and Integration domain (26% of the exam guide), then 1 free question to test it. Part of the AIP-C01 study guide.
AIP-C01 describes an open-weight model that does not fit a managed option, or a team that needs control of the serving stack. The exam tests memory sizing, the causes of slow starts, and whether the team should run SageMaker AI or Kubernetes. Hosting is the last resort when a managed Bedrock option, including Custom Model Import, does not meet the need.
GPU memory
Model weights take roughly the parameter count times the bytes per parameter. A 70 billion parameter model in 16-bit precision needs about 140 GB for weights alone. The KV cache adds memory that grows with batch size and context length, so a size that fits one short request can run out of memory under concurrent long requests. Size for weights plus cache plus overhead.
Large models and tensor parallelism
When the model is too big for one GPU, tensor parallelism splits each layer across several GPUs on one instance. SageMaker AI Large Model Inference (LMI) containers provide serving engines and this configuration, so you set the parallel degree rather than writing the split yourself.
Cold starts
Cold starts are dominated by loading weights into GPU memory. Reduce them by keeping at least one instance warm, loading artifacts from storage close to the endpoint, and avoiding scale-to-zero for interactive GPU endpoints. Scale-to-zero saves money but makes the first user wait for a full model load.
Sharing and shrinking
Inference components let several models share one endpoint, each with its own memory and copy allocation, and scale separately.
Quantization stores weights at lower precision, such as 8-bit or 4-bit, so the model fits a smaller and cheaper instance. Measure quality on your evaluation set before and after.
EKS or SageMaker AI
Choose SageMaker AI when you want managed GPU hosting, deployment safeguards and less cluster work. Choose EKS with GPU nodes when you already run Kubernetes, need custom scheduling, or serve many non-ML workloads on the same cluster.
How to choose
Scenario wording
Choose
“Open-weight model, fully managed, no servers”
Bedrock Custom Model Import, if the model is supported
“Model too big for one GPU”
Tensor parallelism in an LMI container
“First request after idle takes too long”
Keep instances warm, avoid scale-to-zero
“Several small models, one endpoint”
Inference components
“Fit a model on a cheaper instance”
Quantization, validated by evaluation
“Team already runs Kubernetes with GPU nodes”
EKS with GPU nodes
“Small team, managed GPU hosting”
SageMaker AI endpoint
Common traps
Sizing GPU memory for weights only, forgetting the KV cache under concurrency.
Quantizing without testing quality, which can break structured output or accuracy.
One endpoint per small model, which wastes instances. Use inference components.
EKS for a small team with no Kubernetes skills.
Self-hosting a model that Bedrock already serves with the control you need.
A legal research startup wants to host a 70-billion-parameter open-weight model on GPU instances in its own AWS account, because clients require the model to run only inside a dedicated VPC. The model uses FP16 weights, which take about 2 bytes per parameter.
The service must support 16 conversations at the same time, each with a prompt and reply of up to 32,000 tokens. The key-value (KV) cache, which stores attention state for each token in each conversation, grows with both the number of conversations and their length.
The team is choosing the GPU instance type for the endpoint.
Which instance choice meets these requirements?
Choose one answer.
AA. Choose an instance whose total GPU memory is about 140 GB, because the model weights set the minimum memory needed, and leave the KV cache to use whatever memory is left.
BB. Run four replicas of the model on single-GPU instances with 80 GB of memory each, and let the load balancer spread the 16 conversations across them.
CC. Choose a CPU instance with one terabyte of system memory and run the model on the CPU, so that the GPU memory limit no longer applies to the conversations.
DD. Choose an instance with multiple GPUs whose combined memory covers the model weights plus the KV cache for 16 conversations of 32,000 tokens, with some headroom for overhead.
Show the answer and explanation
Answer: D
A. Wrong: The weights alone fill about 140 GB, so the instance would have no room for the KV cache, and long conversations would fail. Memory for 16 conversations of 32,000 tokens must be planned, not left over.
B. Wrong: Each replica must hold the full set of weights, which need about 140 GB, so an 80 GB GPU cannot load the model. Adding replicas does not reduce the memory each one needs.
C. Wrong: The memory fits in system RAM, but generating tokens on the CPU is far slower than on GPUs, so responses would take too long for an interactive service. It also does not meet the latency expectation of a research assistant.
D. Right: The weights and the KV cache both must fit in GPU memory for the model to serve all 16 conversations at full length. A multi-GPU instance with enough combined memory and headroom meets that requirement.
Explanation
The model weights and the KV cache for 16 long conversations both have to fit in GPU memory, and the KV cache can be larger than the weights for long contexts. A multi-GPU instance with enough combined memory and headroom meets that need. Sizing for weights alone, replicating onto small GPUs and moving to CPUs each break a stated constraint.
Want more on Hosting LLMs on containers and GPUs (SageMaker AI, EKS, model loading)? The 5 full-length practice sets have 5 more Hosting LLMs on containers and GPUs (SageMaker AI, EKS, model loading) questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →
Study Hosting LLMs on containers and GPUs (SageMaker AI, EKS, model loading)