AIP-C01 practice question: Hosting LLMs on containers and GPUs (SageMaker AI, EKS, model loading)
Implementation and Integration A free question from our original AIP-C01 practice exam. Try it, then open the answer to see why each option is right or wrong.
A legal research startup wants to host a 70-billion-parameter open-weight model on GPU instances in its own AWS account, because clients require the model to run only inside a dedicated VPC. The model uses FP16 weights, which take about 2 bytes per parameter.
The service must support 16 conversations at the same time, each with a prompt and reply of up to 32,000 tokens. The key-value (KV) cache, which stores attention state for each token in each conversation, grows with both the number of conversations and their length.
The team is choosing the GPU instance type for the endpoint.
Which instance choice meets these requirements?
Choose one answer.
AA. Choose an instance whose total GPU memory is about 140 GB, because the model weights set the minimum memory needed, and leave the KV cache to use whatever memory is left.
BB. Run four replicas of the model on single-GPU instances with 80 GB of memory each, and let the load balancer spread the 16 conversations across them.
CC. Choose a CPU instance with one terabyte of system memory and run the model on the CPU, so that the GPU memory limit no longer applies to the conversations.
DD. Choose an instance with multiple GPUs whose combined memory covers the model weights plus the KV cache for 16 conversations of 32,000 tokens, with some headroom for overhead.
Show the answer and explanation
Answer: D
A. Wrong: The weights alone fill about 140 GB, so the instance would have no room for the KV cache, and long conversations would fail. Memory for 16 conversations of 32,000 tokens must be planned, not left over.
B. Wrong: Each replica must hold the full set of weights, which need about 140 GB, so an 80 GB GPU cannot load the model. Adding replicas does not reduce the memory each one needs.
C. Wrong: The memory fits in system RAM, but generating tokens on the CPU is far slower than on GPUs, so responses would take too long for an interactive service. It also does not meet the latency expectation of a research assistant.
D. Right: The weights and the KV cache both must fit in GPU memory for the model to serve all 16 conversations at full length. A multi-GPU instance with enough combined memory and headroom meets that requirement.
Explanation
The model weights and the KV cache for 16 long conversations both have to fit in GPU memory, and the KV cache can be larger than the weights for long contexts. A multi-GPU instance with enough combined memory and headroom meets that need. Sizing for weights alone, replicating onto small GPUs and moving to CPUs each break a stated constraint.
Want more on Hosting LLMs on containers and GPUs (SageMaker AI, EKS, model loading)? The 5 full-length practice sets have 5 more Hosting LLMs on containers and GPUs (SageMaker AI, EKS, model loading) questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →