AIP-C01 practice question: Operating and tuning vector stores
Operational Efficiency and Optimization for GenAI Applications A free question from our original AIP-C01 practice exam. Try it, then open the answer to see why each option is right or wrong.
A food manufacturer runs a knowledge base for plant staff over about 400,000 document chunks, stored in an Amazon OpenSearch Serverless vector collection. Staff ask questions about sanitation procedures, recall steps and shift handovers.
After the last release added several thousand documents, staff reported that answers now cite the wrong procedures. Nobody noticed until the complaints arrived, because the collection's health and capacity metrics stayed normal.
The team wants early warning when retrieval gets slower or when the right passages stop appearing in the results. The check must run automatically after each release, without a person running manual tests every week.
Which approach meets these requirements?
Choose one answer.
AA. Watch the collection's capacity metrics in CloudWatch, and add more capacity whenever they rise above a set level, on the reasoning that more capacity keeps the retrieved passages accurate.
BB. Count the thumbs-up and thumbs-down clicks that staff give to answers in the interface, and alert the team when the share of thumbs-down clicks rises above a set level each day.
CC. Run a scheduled job that replays labeled staff questions against the vector index, records query latency and recall, publishes both to CloudWatch, and alarms when recall falls.
DD. Raise the maximum capacity of the collection to its largest allowed setting before each release, so that the index can hold the new documents without slowing queries or changing their ranking.
Show the answer and explanation
Answer: C
A. Wrong: Capacity metrics show load, not whether the right chunks are returned, so they would have stayed normal in this incident. Adding capacity does not correct retrieval that returns the wrong procedures. Learn more: What is Amazon OpenSearch Serverless? (opens in a new tab)
B. Wrong: Feedback shows problems only after staff have seen wrong answers, and the clicks are sparse and noisy. It measures satisfaction with answers, not whether retrieval returns the right passages or how quickly it does so.
C. Right: A labeled test set with known correct passages measures recall directly, and the job records latency on every run without manual steps. CloudWatch alarms give the early warning the team needs after each release. Learn more: Using Amazon CloudWatch alarms (opens in a new tab)
D. Wrong: Capacity limits do not guarantee that new documents are ranked well, and this option takes no measurement at all. It spends more on capacity without showing whether retrieval is still correct.
Explanation
A scheduled replay of labeled questions measures recall and latency directly, and publishing the results to CloudWatch lets alarms warn the team after each release. Capacity metrics and user feedback do not show whether the right passages are returned, and raising capacity leaves retrieval quality unmeasured.
Want more on Operating and tuning vector stores? The 5 full-length practice sets have 5 more Operating and tuning vector stores questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →