Connecting data sources and keeping vector stores current: AIP-C01 study notes and practice questions
What to know about Connecting data sources and keeping vector stores current for the AIP-C01 exam, from the Foundation Model Integration, Data Management, and Compliance domain (31% of the exam guide), then 1 free question to test it. Part of the AIP-C01 study guide.
Typical scenarios: a SharePoint library changes every hour, an internal system has no connector, deleted contracts still appear in answers, or a critical update must be searchable within seconds. Choose the connector and the refresh method that match the freshness need.
Built-in connectors first
Managed knowledge bases include connectors for S3, Confluence, SharePoint, the web crawler, Google Drive and OneDrive, plus a custom connector for other systems. Use them before writing ingestion code. Amazon Kendra is closed to new customers, so a new enterprise search build uses a managed knowledge base instead.
Sync jobs and incremental change
A sync (ingestion job) processes only added, modified or deleted documents since the last run. The first run processes everything.
Deleted documents are removed from the vector store during sync, so stale chunks stop being retrieved. Until a sync runs, the deleted content can still be returned.
Schedule syncs with EventBridge Scheduler, or start one from an event. An S3 event notification can invoke Lambda, which calls the ingestion API when a file arrives.
Direct ingestion for immediate updates
Direct ingestion (IngestKnowledgeBaseDocuments) pushes document content into a knowledge base without a sync job, so a change is searchable almost at once. Removal uses DeleteKnowledgeBaseDocuments. Use it for near-real-time needs on a custom data source. You then own change tracking, so do not add it where a sync already works.
Unsupported systems
Export records from the system to S3 on a schedule, one file per document with a metadata sidecar. Point the knowledge base at that S3 source and sync. This keeps one pipeline and handles deletions through the export diff.
How to choose
Scenario wording
Choose
“SharePoint must refresh hourly”
Built-in connector with a scheduled sync
“Changes searchable within seconds, custom source”
Direct ingestion
“Index each new S3 file as it lands”
S3 event to Lambda that starts an ingestion job
“Deleted contracts must stop appearing”
Sync with deletion handling, or direct deletion
“No connector for an internal system”
Scheduled export to S3 with metadata, then sync
Common traps
Rebuilding a connector that already exists as a built-in.
Full re-ingestion on every change, instead of incremental sync.
Assuming deletion is automatic without a sync run.
Using direct ingestion for a source where a sync already tracks changes, which adds work for no benefit.
Expecting a scheduled sync to meet a seconds-level requirement.
A software company's support assistant must answer from three sources: product pages on its public documentation website, runbooks in a Confluence space, and contracts in a SharePoint library.
The documentation team edits pages several times a day, and deleted pages must also disappear from the assistant's answers. The platform team has no spare engineers to build or run custom data pipelines.
The index should refresh each night, and each sync should copy only the changes.
Which solution meets these requirements with the LEAST custom development?
Choose one answer.
AA. Add the website, Confluence and SharePoint as data sources of one knowledge base, using the built-in web crawler and connectors, and schedule incremental syncs each night.
BB. Write a nightly script for each system that exports its pages to an S3 bucket, and point a single S3 data source at that bucket so that the knowledge base syncs from the exports.
CC. Create an AWS Glue crawler over each content system and point the knowledge base at the crawler output, so the Data Catalog keeps the documents current.
DD. Build a Step Functions workflow that calls each system's API every night, embeds every page with a Titan embeddings call, and writes the vectors to a self-managed index.
Show the answer and explanation
Answer: A
A. Right: The built-in connectors read each source directly, and incremental sync picks up added, changed and deleted content. No pipeline code is needed, which meets the freshness, deletion and effort requirements. Learn more: Crawl web pages for your knowledge base (opens in a new tab)
B. Wrong: The scripts would need to be written, monitored and fixed whenever an API changes, which the team cannot support. Deletions would also depend on the export script noticing removed pages.
C. Wrong: A Glue crawler builds Data Catalog table metadata from stored data, and it does not extract page text from Confluence or SharePoint. The content would not reach the knowledge base.
Built-in web crawler and SharePoint and Confluence connectors read each source directly, and incremental sync handles changes and deletions on a nightly schedule. Exporting scripts, a Glue crawler and a custom workflow each add development work or fail to load the content at all.
One knowledge base with three data sources, synced each night, answers from the website, Confluence and SharePoint together.
Want more on Connecting data sources and keeping vector stores current? The 5 full-length practice sets have 5 more Connecting data sources and keeping vector stores current questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →
Study Connecting data sources and keeping vector stores current