AIP-C01 practice question: Latency optimization (streaming, latency-optimized inference, parallel calls)
Operational Efficiency and Optimization for GenAI Applications A free question from our original AIP-C01 practice exam. Try it, then open the answer to see why each option is right or wrong.
A payments company runs an assistant for its customer service agents. While a customer is on the phone, the agent types a question about a disputed charge, and the assistant drafts a reply that the agent reads aloud.
The assistant uses a large model on Amazon Bedrock. Agents complain that the first words take several seconds to appear during peak hours, which makes calls awkward. Overnight, the company also runs a batch job that writes weekly account summaries, and nobody waits for those results.
The interactive calls must get the fastest response they can at peak, the company wants no change to the model or the prompts, and the batch job should cost as little as possible.
Which configuration meets these requirements?
Choose one answer.
AA. Buy Provisioned Throughput with one model unit for the assistant's model, and send all calls, both interactive and batch, to that reserved capacity so that no request queues behind other customers.
BB. Move the agent assistant to the batch inference API, which writes each draft to an Amazon S3 bucket that the agents can read after the call ends, and keep the weekly summaries on the same API.
CC. Send the interactive calls with the Priority service tier and the overnight summaries with the Flex service tier, keeping the same model and the same prompts.
DD. Deploy a copy of the model on a SageMaker AI real-time endpoint with the largest GPU instance, and point the agent application to that endpoint for all calls and for the batch summaries.
Show the answer and explanation
Answer: C
A. Wrong: Reserved capacity reduces queuing at peaks, but its hourly charge runs all day, including the nights when only the batch job runs. The batch job would also pay that reserved price, which does not meet the cost requirement. Learn more: Provisioned Throughput for Amazon Bedrock (opens in a new tab)
D. Wrong: A self-managed endpoint adds instance hours, scaling work and a model copy to maintain. Running the overnight summaries on the largest instance also raises their cost, so the option fails the cost requirement.
Explanation
Service tiers are set per request, so the interactive calls can use the Priority tier for faster responses at peak while the overnight summaries use the Flex tier at lower cost, with the same model and prompts. Reserved capacity charges all day, batch inference is asynchronous, and a self-managed endpoint adds operational work.
Want more on Latency optimization (streaming, latency-optimized inference, parallel calls)? The 5 full-length practice sets have 5 more Latency optimization (streaming, latency-optimized inference, parallel calls) questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →