Prompt caching and semantic caching: AIP-C01 study notes and practice questions
What to know about Prompt caching and semantic caching for the AIP-C01 exam, from the Operational Efficiency and Optimization for GenAI Applications domain (12% of the exam guide), then 1 free question to test it. Part of the AIP-C01 study guide.
Know which cache sits where in the request path. Prompt caching works inside the model call and lowers the cost of repeated input. Semantic and exact-match caches sit in front of the model and can skip the call entirely.
Bedrock prompt caching
Prompt caching stores the processed prefix of a prompt, such as a long system prompt, a reference document or tool definitions. You add cache checkpoints to the request. Later calls with an identical prefix read those tokens at a lower price and with lower latency. The prefix must match exactly, so put static content first and the user’s question last. The order is tools, then the system prompt, then messages, and a checkpoint caches everything before it. Cache entries expire after a few minutes without use.
Prompt caching lowers input cost and latency. It does not reduce output tokens.
Application-level caches
A semantic cache stores question and answer pairs. It embeds each question, searches the stored embeddings with vector search in a service such as Amazon ElastiCache or MemoryDB, and returns a stored answer when a new question is close enough. Set the similarity threshold high, since a loose match returns the wrong answer. An exact-match cache hashes the normalized request and uses the hash as the key in DynamoDB. It is simpler and safe for identical repeats. CloudFront can cache a public answer at the edge when every user gets the same response.
Invalidation
Cached answers go stale when source documents change. Tie each cache entry to a document version or content hash, and delete entries when the knowledge base syncs. Prompt cache entries expire on their own, but a semantic or exact-match cache keeps serving an old answer until you remove it.
How to choose
Scenario wording
Choose
The same long system prompt or document in every call
Bedrock prompt caching
Many users ask the same question in different words
Semantic cache with vector search
Identical requests repeat exactly
Exact-match hash cache in DynamoDB
One public answer served to all users worldwide
CloudFront edge caching
Source documents change and answers must stay current
Cache keyed by document version, invalidated on sync
Common traps
Expecting prompt caching to cut output tokens. It only affects repeated input.
Putting the user’s question before the static document, so the prefix never matches.
Caching personalized answers at the edge, which can show one user’s answer to another.
A loose semantic match that returns the answer to a different question.
No invalidation, so the cache keeps serving a superseded policy.
A commercial real estate brokerage runs an assistant that answers analysts' questions about a single 60,000-token master lease agreement. The system prompt and the agreement text are the same for every question about that lease, and analysts ask dozens of questions in a session.
Each call sends the full prefix again, so input costs and response times are high. Only the question at the end of each request changes, and the brokerage wants answer quality to stay exactly as it is now.
The change must cut the input cost and latency of the repeated prefix while keeping the full agreement in every request. It must not change the model.
Which change meets these requirements with the LEAST code?
Choose one answer.
AA. Store each question and its answer in Amazon DynamoDB keyed by the question text, and return the stored answer whenever an analyst repeats the same question word for word.
BB. Add a cache checkpoint after the system prompt and agreement text in the Converse request, so later calls reuse the cached prefix, with the analyst's question placed after it.
CC. Lower the temperature to 0 so that the model returns the same answer for each repeated question, and keep sending the full agreement in every request, as the current design does.
DD. Split the agreement into fifty smaller sections, and send each question with only the section it seems to need, so that each request carries far fewer input tokens than the full document.
Show the answer and explanation
Answer: B
A. Wrong: Exact-match storage helps only when a question is asked again word for word, and analysts rarely repeat questions exactly. Each new question still sends the full agreement, so the input cost stays high.
B. Right: Prompt caching stores the repeated prefix, so cached input tokens cost less and are processed faster. The prefix must match exactly, which is why the question goes after the checkpoint, and the model and answers stay the same. Learn more: Prompt caching for faster model inference (opens in a new tab)
C. Wrong: Temperature affects randomness in the output, not the cost of the input tokens sent with each request. The agreement is still billed in full on every call, so the cost and latency targets are not met. Learn more: Inference request parameters (opens in a new tab)
D. Wrong: Smaller prompts cut input tokens, but a question about a clause in another section can be answered wrongly, which changes answer quality. The option also breaks the requirement to keep the full agreement in every request.
Explanation
Prompt caching lets the repeated system prompt and agreement text be reused at lower cost and with faster processing, as long as the prefix matches exactly. Putting the varying question after the cache checkpoint keeps the prefix stable. Exact-match answer storage, lower temperature and section splitting either miss the input cost or change answer quality.
Want more on Prompt caching and semantic caching? The 5 full-length practice sets have 5 more Prompt caching and semantic caching questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →