Troubleshooting retrieval quality: AIP-C01 study notes and practice questions
What to know about Troubleshooting retrieval quality for the AIP-C01 exam, from the Testing, Validation, and Troubleshooting domain (11% of the exam guide), then 1 free question to test it. Part of the AIP-C01 study guide.
Work backward from the symptom through the pipeline: source, parsing, chunking, embedding, index, then query filters. Most retrieval faults start in one of the earlier stages.
Right document not retrieved
If a relevant document never appears, check chunking first. Chunks that are too large dilute their meaning and may not match the question. Chunks that are too small lose context. Hierarchical chunking matches on small child chunks and returns the parent chunk for context, which handles both problems. Then check the query. A question phrased with different words may need hybrid search or query rewriting.
Embedding mismatch
Documents and queries must use the same embedding model, with the same dimensions and normalization settings. After you change the embedding model, re-embed every stored vector and rebuild the index. A mixed index returns poor similarity scores without raising an error.
Stale content and sync
If answers use old content, check the sync job. A knowledge base sync ingests only added, changed or deleted documents. A failed or unscheduled job leaves stale chunks in place. Check the ingestion job status and its failure reasons. For urgent updates, use direct ingestion to push the change immediately. Deleted documents must be removed too, or their chunks keep answering.
Metadata filters and parsing
A metadata filter that excludes valid documents often has a typo, a wrong value type, or a missing attribute on older documents. Check the metadata sidecar files and the filter values. Tables and figures in PDFs can lose their structure under default parsing. A foundation model parser or Amazon Bedrock Data Automation keeps tables and figures more reliably.
Slow vector search
Slow search at scale points to the index and the store. Check the index settings and shard sizing (see vector store operations). Filter before ranking where the store supports it, and request fewer results. Rerank only the top set.
A law firm's knowledge base holds about 40,000 contract templates and clause libraries. Attorneys ask questions such as which template contains a specific indemnity cap. The knowledge base uses fixed-size chunks of about 2,000 tokens.
For these questions, the right template is often missing from the retrieved results, even though it contains the exact clause. Each chunk holds several unrelated clauses, so the clause's wording is diluted in the chunk's embedding. The firm wants answers in under three seconds and does not want to add a new search service.
Which change is most likely to fix the retrieval?
Choose one answer.
AA. Increase the number of results retrieved per query from 5 to 100, so the model receives many more chunks and can find the clause among them.
BB. Re-ingest the templates with fixed-size chunks of about 300 tokens with some overlap, so each clause has its own embedding.
CC. Increase the chunk size to 8,000 tokens so that each template is held in fewer chunks, and each document keeps its full context in one embedding.
DD. Switch the generation model to one with a larger context window, so that it can find the indemnity cap in whichever template the retriever returns.
Show the answer and explanation
Answer: B
A. Wrong: Retrieving 100 chunks slows each answer and sends far more text to the model, which raises cost. The right clause still has to rank high enough to be retrieved in the first place.
B. Right: Smaller chunks keep each clause's meaning from being diluted by unrelated text, so the right template ranks higher. The change uses the same knowledge base and adds no service, and latency stays low. Learn more: Chunking strategies (opens in a new tab)
C. Wrong: Larger chunks dilute the clause further, because one embedding must represent many more unrelated sentences. This makes the retrieval problem worse, not better.
D. Wrong: The generation model can only use what retrieval returns, so the missing template stays missing. A larger context window does not change which chunks are retrieved.
Explanation
Retrieval can miss an exact clause when a large chunk's embedding is dominated by unrelated text. Re-chunking the templates into smaller pieces lets each clause carry its own meaning, so the right template ranks higher, with no new service. Retrieving more chunks, bigger chunks and a larger generation model each leave the retrieval problem in place or make it worse.
Want more on Troubleshooting retrieval quality? The 5 full-length practice sets have 5 more Troubleshooting retrieval quality questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →