Operating and tuning vector stores: AIP-C01 study notes and practice questions
What to know about Operating and tuning vector stores for the AIP-C01 exam, from the Operational Efficiency and Optimization for GenAI Applications domain (12% of the exam guide), then 1 free question to test it. Part of the AIP-C01 study guide.
Vector store questions are operational. You are told that a store has become slow, inaccurate or costly after data growth or a bulk load, and you must pick the fix that addresses the cause.
Monitor speed and recall
Track query latency percentiles and a recall measure. Recall is measured by comparing approximate nearest-neighbor results with exact results on a test set, or through retrieval evaluation with a golden set of questions and expected sources. A fast store that misses the right chunk is failing, even if its latency looks good.
Tune HNSW parameters
HNSW index settings trade speed for accuracy. The parameter m sets the number of connections per node. More connections raise recall and memory use. ef_construction controls index build quality. ef_search controls how many candidates a query examines: a higher value improves recall and slows queries. Because m and ef_construction are fixed when the index is built, changing them means a rebuild. ef_search can be changed without a rebuild, so tune it first.
Bulk loads and capacity
A large bulk load can leave many small index segments, which slows queries. On a managed OpenSearch domain, force merge after the load to reduce segment count, and run it off-peak. A serverless collection manages segments for you. OpenSearch Serverless scales in OpenSearch Compute Units (OCUs). Watch OCU usage and search and indexing times, and raise capacity limits if scale is the bottleneck. On managed clusters, size shards to a practical size and plan the shard count for expected growth.
Validate stored chunks
Bad data produces bad retrieval. Check each chunk after ingestion: text is not empty, metadata fields are present, the embedding dimension matches the model, chunks are not duplicated, and document IDs let you delete stale chunks. Run a scheduled job that samples stored records and alerts on failures.
How to choose
Scenario wording
Choose
Queries slowed down after a bulk load
Force merge on managed OpenSearch, or check segment count
Fast queries that miss relevant chunks
Raise ef_search and measure recall
Index build quality or memory is the problem
Review m and ef_construction, then rebuild
Serverless throughput or cost is limiting
Review OCU usage and adjust capacity limits
Store has outgrown its shard plan
Re-plan shard count and reindex
Sampled chunks are malformed
Validate at ingestion and re-ingest
Common traps
Raising m to speed up queries. A higher m raises memory use and build time, and it needs a rebuild.
Tuning ef_search without a recall test, so you cannot tell whether it helped.
Ignoring stale chunks left behind after documents are deleted.
Changing the embedding model without re-embedding the whole corpus.
Oversized shards, which slow recovery and queries.
A food manufacturer runs a knowledge base for plant staff over about 400,000 document chunks, stored in an Amazon OpenSearch Serverless vector collection. Staff ask questions about sanitation procedures, recall steps and shift handovers.
After the last release added several thousand documents, staff reported that answers now cite the wrong procedures. Nobody noticed until the complaints arrived, because the collection's health and capacity metrics stayed normal.
The team wants early warning when retrieval gets slower or when the right passages stop appearing in the results. The check must run automatically after each release, without a person running manual tests every week.
Which approach meets these requirements?
Choose one answer.
AA. Watch the collection's capacity metrics in CloudWatch, and add more capacity whenever they rise above a set level, on the reasoning that more capacity keeps the retrieved passages accurate.
BB. Count the thumbs-up and thumbs-down clicks that staff give to answers in the interface, and alert the team when the share of thumbs-down clicks rises above a set level each day.
CC. Run a scheduled job that replays labeled staff questions against the vector index, records query latency and recall, publishes both to CloudWatch, and alarms when recall falls.
DD. Raise the maximum capacity of the collection to its largest allowed setting before each release, so that the index can hold the new documents without slowing queries or changing their ranking.
Show the answer and explanation
Answer: C
A. Wrong: Capacity metrics show load, not whether the right chunks are returned, so they would have stayed normal in this incident. Adding capacity does not correct retrieval that returns the wrong procedures. Learn more: What is Amazon OpenSearch Serverless? (opens in a new tab)
B. Wrong: Feedback shows problems only after staff have seen wrong answers, and the clicks are sparse and noisy. It measures satisfaction with answers, not whether retrieval returns the right passages or how quickly it does so.
C. Right: A labeled test set with known correct passages measures recall directly, and the job records latency on every run without manual steps. CloudWatch alarms give the early warning the team needs after each release. Learn more: Using Amazon CloudWatch alarms (opens in a new tab)
D. Wrong: Capacity limits do not guarantee that new documents are ranked well, and this option takes no measurement at all. It spends more on capacity without showing whether retrieval is still correct.
Explanation
A scheduled replay of labeled questions measures recall and latency directly, and publishing the results to CloudWatch lets alarms warn the team after each release. Capacity metrics and user feedback do not show whether the right passages are returned, and raising capacity leaves retrieval quality unmeasured.
Want more on Operating and tuning vector stores? The 5 full-length practice sets have 5 more Operating and tuning vector stores questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →