Throughput, quotas and capacity planning: AIP-C01 study notes and practice questions
What to know about Throughput, quotas and capacity planning for the AIP-C01 exam, from the Operational Efficiency and Optimization for GenAI Applications domain (12% of the exam guide), then 1 free question to test it. Part of the AIP-C01 study guide.
AIP-C01 asks how to keep a generative AI workload responsive as traffic grows, without throttling errors and without paying for idle capacity. Most questions turn on one number: the quota that peak traffic would exceed.
Quotas and request rate
Bedrock on-demand quotas limit both requests per minute and tokens per minute for each model in each Region. Estimate peak load as requests per minute multiplied by the average input plus output tokens per request, then compare the result with the tokens-per-minute quota. When a request starts, Bedrock reserves its input tokens plus maxTokens against the quota and adjusts after the response, so a maxTokens far above real need lowers effective throughput. Some newer models also count output tokens at a higher burndown rate. Check the quotas in Service Quotas and request an increase before launch if the gap is large. Quota hits appear as ThrottlingException or HTTP 429.
Provisioned Throughput
Provisioned Throughput reserves model units for a model at a fixed hourly commitment. It gives steady capacity for predictable, high-volume traffic and is required for some custom models. Size the model units from measured load, not from a guess at peak. For spiky or low-volume traffic, on-demand is cheaper because you pay for idle units with Provisioned Throughput.
Smooth bursts and control concurrency
Put requests on an Amazon SQS queue and let workers consume them at a rate the quota allows. Bursts then wait rather than fail. Retry throttled calls with exponential backoff and jitter, which the AWS SDKs do by default. Cross-Region inference spreads bursts across more capacity: a geographic profile stays inside one geography, and a global profile reaches the most capacity.
Backlogs and self-hosted endpoints
A large backlog with no per-item deadline fits batch inference, which runs outside the interactive quota. For models you host on SageMaker AI, use endpoint auto scaling on a load metric such as invocations per instance, and set a maximum instance count so cost has a ceiling. Choose asynchronous inference for long or queued requests.
How to choose
Scenario wording
Choose
Steady high volume that must always have capacity
Provisioned Throughput
Spiky traffic with a low average
On-demand with an SQS queue and backoff
Backlog to finish overnight, no per-item deadline
Batch inference
Burst beyond one Region’s quota, residency allows it
Global cross-Region inference profile
Burst that must stay inside one geography
Geographic cross-Region inference profile
Self-hosted endpoint slows under load
SageMaker AI auto scaling on a load metric
Common traps
Buying Provisioned Throughput for spiky traffic, which pays for idle model units.
Retrying at once without backoff, which keeps the throttle hot.
Planning only on requests per minute and ignoring tokens per minute.
Global profiles for residency-bound data, since requests can route outside the geography.
Forgetting to request a quota increase before launch, so the first peak is throttled.
A marketing agency runs a campaign tool that generates product descriptions for about 60 retail clients. Each request includes a long brand guide, so most requests carry several thousand input tokens, and the tool calls a model on Amazon Bedrock in one Region.
In a load test a month before launch, the tool sends about 300 requests a minute in short bursts. Many calls fail with throttling errors, even though the request count is well below what the team expected to be the limit. The launch will last only a week, and the budget is tight.
The fix must remove most of the throttling during the launch, avoid paying for capacity the team will not need after the week, and keep the workload on the same model.
Which approach meets these requirements?
Choose one answer.
AA. Estimate tokens per minute from real prompt and response lengths at the peak rate, request a quota increase with headroom, and retry throttled calls with backoff.
BB. Move the requests to Provisioned Throughput with the largest number of model units for the whole month, so that capacity is reserved and throttling cannot occur during the launch.
CC. Set the client timeout to five minutes and retry throttled calls at a fixed interval, so that each request waits in line until quota frees up, without changing the quota.
DD. Split the tool into one Lambda function per client, because each function gets its own default quota, and send each client's requests from its own function during the launch.
Show the answer and explanation
Answer: A
A. Right: Bedrock throttles on tokens per minute as well as requests per minute, so long brand-guide prompts can hit the quota at modest request rates. Sizing tokens per minute from real lengths and requesting an increase a month ahead addresses the cause with no capacity commitment, and backoff smooths the bursts. Learn more: Request an increase for Amazon Bedrock runtime quotas (opens in a new tab)
B. Wrong: Reserved capacity removes throttling, but the hourly commitment would run for the whole month, far beyond the one-week launch. This conflicts with the budget and with the requirement to avoid paying for unneeded capacity. Learn more: Provisioned Throughput for Amazon Bedrock (opens in a new tab)
D. Wrong: Quotas apply to the account and Region, not to each function, so splitting the tool does not add quota. The option changes the architecture without solving the throttling. Learn more: Amazon Bedrock quotas (opens in a new tab)
Explanation
Bedrock throttles on tokens per minute as well as requests per minute, so large brand-guide prompts can exhaust the quota at modest request rates. Sizing tokens per minute from real lengths, requesting a quota increase well before launch and retrying with backoff address the cause at on-demand prices. Monthly reserved capacity, longer timeouts and per-function splitting do not fix the quota.
Want more on Throughput, quotas and capacity planning? The 5 full-length practice sets have 5 more Throughput, quotas and capacity planning questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →