Latency optimization (streaming, latency-optimized inference, parallel calls): AIP-C01 study notes and practice questions
What to know about Latency optimization (streaming, latency-optimized inference, parallel calls) for the AIP-C01 exam, from the Operational Efficiency and Optimization for GenAI Applications domain (12% of the exam guide), then 1 free question to test it. Part of the AIP-C01 study guide.
Latency has several parts: time to first token, total generation time, retrieval, and tool calls. The exam expects you to find the slow part and fix that part, not to add a bigger model or a cache everywhere.
Stream the response
Streaming sends tokens as they are generated, so time to first token drops even when total time stays the same. Use ConverseStream or InvokeModelWithResponseStream. In a browser chat, send the stream through a WebSocket API or a Lambda response stream. A REST API integration times out after 29 seconds by default, which long generations can exceed. Streaming does not help a background job that nobody watches.
Faster inference options
Latency-optimized inference runs some models on faster hardware. The Priority service tier serves a request ahead of Standard and Flex traffic, at a higher price. Moving an interactive step to a smaller, faster model is often the biggest gain. Benchmark candidates on your own prompts, because speed varies by model and by prompt size.
Parallel and precomputed work
Run independent subtasks in parallel, such as searching two sources or running a classifier beside a summarizer. Use Step Functions Map states or concurrent calls in Lambda. Keep dependent steps in order. For predictable questions, such as FAQs or fixed product summaries, precompute the answers and serve them from a cache.
Retrieval latency
Retrieval is often the slow step. Retrieve fewer results, filter by metadata before ranking, and keep the vector store in the same Region as the application and the model. Tune the vector index for query speed. Reranking adds time, so apply it only when the quality gain is worth the wait.
How to choose
Scenario wording
Choose
Users wait too long before any text appears
Streaming with ConverseStream
Latency-critical customer chat where cost matters less
Priority tier or latency-optimized inference
Independent lookups delay the answer
Run them in parallel and merge the results
The same predictable questions arrive often
Precompute and cache
Retrieval dominates the time
Fewer results, metadata filters, same-Region store
Long generations fail at 29 seconds through REST
REST API response streaming, a WebSocket API or Lambda response streaming
Common traps
Streaming to fix a slow backend. It lowers time to first token, not the work the model does.
Parallelizing dependent steps, which breaks the order the answer needs.
Using the largest model for an interactive step where a smaller model meets the quality bar.
Batch inference for a live chat. Batch is for offline jobs with no real-time need.
Adding reranking everywhere when the slow part is somewhere else.
A payments company runs an assistant for its customer service agents. While a customer is on the phone, the agent types a question about a disputed charge, and the assistant drafts a reply that the agent reads aloud.
The assistant uses a large model on Amazon Bedrock. Agents complain that the first words take several seconds to appear during peak hours, which makes calls awkward. Overnight, the company also runs a batch job that writes weekly account summaries, and nobody waits for those results.
The interactive calls must get the fastest response they can at peak, the company wants no change to the model or the prompts, and the batch job should cost as little as possible.
Which configuration meets these requirements?
Choose one answer.
AA. Buy Provisioned Throughput with one model unit for the assistant's model, and send all calls, both interactive and batch, to that reserved capacity so that no request queues behind other customers.
BB. Move the agent assistant to the batch inference API, which writes each draft to an Amazon S3 bucket that the agents can read after the call ends, and keep the weekly summaries on the same API.
CC. Send the interactive calls with the Priority service tier and the overnight summaries with the Flex service tier, keeping the same model and the same prompts.
DD. Deploy a copy of the model on a SageMaker AI real-time endpoint with the largest GPU instance, and point the agent application to that endpoint for all calls and for the batch summaries.
Show the answer and explanation
Answer: C
A. Wrong: Reserved capacity reduces queuing at peaks, but its hourly charge runs all day, including the nights when only the batch job runs. The batch job would also pay that reserved price, which does not meet the cost requirement. Learn more: Provisioned Throughput for Amazon Bedrock (opens in a new tab)
D. Wrong: A self-managed endpoint adds instance hours, scaling work and a model copy to maintain. Running the overnight summaries on the largest instance also raises their cost, so the option fails the cost requirement.
Explanation
Service tiers are set per request, so the interactive calls can use the Priority tier for faster responses at peak while the overnight summaries use the Flex tier at lower cost, with the same model and prompts. Reserved capacity charges all day, batch inference is asynchronous, and a self-managed endpoint adds operational work.
Want more on Latency optimization (streaming, latency-optimized inference, parallel calls)? The 5 full-length practice sets have 5 more Latency optimization (streaming, latency-optimized inference, parallel calls) questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →
Study Latency optimization (streaming, latency-optimized inference, parallel calls)