AIP-C01 practice question: Troubleshooting retrieval quality
Testing, Validation, and Troubleshooting A free question from our original AIP-C01 practice exam. Try it, then open the answer to see why each option is right or wrong.
A law firm's knowledge base holds about 40,000 contract templates and clause libraries. Attorneys ask questions such as which template contains a specific indemnity cap. The knowledge base uses fixed-size chunks of about 2,000 tokens.
For these questions, the right template is often missing from the retrieved results, even though it contains the exact clause. Each chunk holds several unrelated clauses, so the clause's wording is diluted in the chunk's embedding. The firm wants answers in under three seconds and does not want to add a new search service.
Which change is most likely to fix the retrieval?
Choose one answer.
AA. Increase the number of results retrieved per query from 5 to 100, so the model receives many more chunks and can find the clause among them.
BB. Re-ingest the templates with fixed-size chunks of about 300 tokens with some overlap, so each clause has its own embedding.
CC. Increase the chunk size to 8,000 tokens so that each template is held in fewer chunks, and each document keeps its full context in one embedding.
DD. Switch the generation model to one with a larger context window, so that it can find the indemnity cap in whichever template the retriever returns.
Show the answer and explanation
Answer: B
A. Wrong: Retrieving 100 chunks slows each answer and sends far more text to the model, which raises cost. The right clause still has to rank high enough to be retrieved in the first place.
B. Right: Smaller chunks keep each clause's meaning from being diluted by unrelated text, so the right template ranks higher. The change uses the same knowledge base and adds no service, and latency stays low. Learn more: Chunking strategies (opens in a new tab)
C. Wrong: Larger chunks dilute the clause further, because one embedding must represent many more unrelated sentences. This makes the retrieval problem worse, not better.
D. Wrong: The generation model can only use what retrieval returns, so the missing template stays missing. A larger context window does not change which chunks are retrieved.
Explanation
Retrieval can miss an exact clause when a large chunk's embedding is dominated by unrelated text. Re-chunking the templates into smaller pieces lets each clause carry its own meaning, so the right template ranks higher, with no new service. Retrieving more chunks, bigger chunks and a larger generation model each leave the retrieval problem in place or make it worse.
Want more on Troubleshooting retrieval quality? The 5 full-length practice sets have 5 more Troubleshooting retrieval quality questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →