Glue Data Catalog and crawlers: DEA-C01 study notes and practice questions
What to know about Glue Data Catalog and crawlers for the DEA-C01 exam, from the Data Store Management domain (26% of the exam guide), then 2 free questions to test it. Part of the DEA-C01 study guide.
Create a single schema for each S3 path (table grouping setting) stops a crawler making one table per folder when files are compatible.
Schema change policy: update the table, add new columns only, or log changes and ignore.
Incremental crawls (crawl new folders only, or S3 event mode) save time and cost.
JDBC sources need a Glue connection with the VPC subnet, a security group that allows the database and a self-referencing rule, and credentials (Secrets Manager).
Alternatives to crawling
Partition projection in Athena calculates partitions from a pattern (such as dates) with no catalog updates.
MSCK REPAIR TABLE or ALTER TABLE ADD PARTITION in Athena.
Glue API / CloudFormation to define tables directly.
Glue jobs can update the catalog as they write (enableUpdateCatalog).
Business metadata
Amazon SageMaker Catalog (in SageMaker Unified Studio, the evolution of Amazon DataZone) adds business glossaries, data products and governed sharing on top of technical metadata.
Common traps
A crawler creating hundreds of tables from one dataset: turn on single schema per path, or check for incompatible files.
New partitions not visible in Athena: run a crawler, add partitions, or use partition projection.
A dental chain's data lake stores one folder for each clinic under a shared S3 prefix. Every clinic's folder holds the same Parquet table, but a crawler splits the folders into dozens of separate catalog tables because it treats each folder as a different schema.
The analysts want one table with the clinic as a partition, and the crawler should keep finding new clinic folders each night.
Which crawler setting meets these requirements?
Choose one answer.
AA. Create one crawler for each clinic folder, and point each crawler at its own table name so that the catalog stays organized.
BB. Turn off the crawler's schedule, and delete the extra tables by hand after each run, keeping the table with the most rows.
CC. Set the crawler to create a single schema for each S3 path, so folders with compatible schemas map to one partitioned table.
DD. Rename each clinic folder so that the crawler sees the same name, which makes the crawler merge the folders into one table.
Show the answer and explanation
Answer: C
A. Wrong: One crawler per folder multiplies the crawlers to maintain and still creates a table per clinic. It also breaks the goal of a single table that picks up new folders automatically.
B. Wrong: Manual cleanup after every run is error-prone, and turning off the schedule stops the crawler from finding new clinic folders. The option fails both requirements.
C. Right: Correct. This crawler setting groups compatible S3 paths into one table, and the folder layout becomes partitions. New clinic folders with the same schema are added to the same table on the next run. Learn more: Customizing crawler behavior (opens in a new tab)
D. Wrong: Renaming folders does not change the schemas the crawler compares, and the folder names still differ. The option does not address the grouping behavior.
Explanation
A crawler groups compatible files into one table only when it is told to create a single schema for each S3 path. With that setting, clinic folders that share a schema map to one table, and the folder becomes a partition. Per-folder crawlers, manual cleanup and renaming folders all add work or miss the grouping setting.
A dental chain runs a patient-records database on SQL Server in its own data center. The data team wants a Glue crawler to read the schema of that database into the Data Catalog, so that Athena and Glue jobs can use it.
The crawler fails with connection timeouts. The network team has connected the data center to the AWS VPC with AWS Direct Connect, and the database user's password is stored in AWS Secrets Manager.
Which configuration allows the crawler to connect?
Choose one answer.
AA. Open the SQL Server port on the data center firewall to the entire internet, and point the crawler at the public address.
BB. Export the schema from SQL Server as a DDL script each night, and upload the script to S3 so that the crawler can read it as a file.
CC. Place the crawler in a public subnet with an internet gateway, and put the database password in the crawler's environment variables.
DD. Create a Glue connection with the password secret and a VPC subnet and security group that route to the data center.
Show the answer and explanation
Answer: D
A. Wrong: Opening a database to the internet exposes patient records to threats and breaks the network design. The option does not fit the data protection requirements for this data.
B. Wrong: A DDL export works for the schema, but it is a manual pipeline that the crawler is not needed for. The team asked for the crawler to read the database, so the option changes the design.
C. Wrong: A public subnet does not route to the data center, and environment variables are not how Glue passes credentials. The option stores the password in an unsafe place and still fails to route.
D. Right: Correct. A Glue connection holds the JDBC endpoint, credentials and the VPC subnet and security group the crawler uses. The subnet's route table sends traffic to the data center over Direct Connect, and the security group allows the database port. Learn more: Connecting to data (opens in a new tab)
Explanation
A Glue connection is the object that tells crawlers and jobs how to reach a database inside a VPC. It names the subnet and security group, so the crawler uses the Direct Connect route to the data center, and it refers to the password in Secrets Manager. Public exposure, schema exports and public subnets each break security or routing.
Want more on Glue Data Catalog and crawlers? The 5 full-length practice sets have 8 more Glue Data Catalog and crawlers questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →