File formats and conversion: DEA-C01 study notes and practice questions
What to know about File formats and conversion for the DEA-C01 exam, from the Data Ingestion and Transformation domain (34% of the exam guide), then 2 free questions to test it. Part of the DEA-C01 study guide.
An esports league sends player-telemetry events as JSON records through Amazon Data Firehose into Amazon S3. Analysts query the data with Amazon Athena, and the team wants the files delivered as Parquet, not JSON.
The team does not want to run an extra conversion job after delivery, and it wants the change to stay within the existing delivery stream.
Which change meets these requirements?
Choose one answer.
AA. Turn on gzip compression for the delivery stream, so that the JSON files in S3 are smaller and cheaper to scan.
BB. Add an AWS Lambda transformation to the delivery stream that writes each record as a separate Parquet file in S3.
CC. Create a Glue Data Catalog table for the event schema, and turn on Parquet record format conversion in the delivery stream.
DD. Set a larger buffer size on the delivery stream, and run an Athena CTAS query after each delivery to write Parquet copies.
Show the answer and explanation
Answer: C
A. Wrong: Gzip compresses the JSON but keeps the row-based JSON format. The team asked for Parquet, so this option does not meet the requirement.
B. Wrong: A Lambda transformation could produce Parquet, but writing one file per record creates a huge number of small objects. The option is also more custom code than the team wants.
C. Right: Correct. Firehose converts JSON to Parquet or ORC using the schema in a Glue Data Catalog table, and it converts each record before writing to S3. No separate job runs after delivery, and the existing stream stays in place. Learn more: Enable record format conversion (opens in a new tab)
D. Wrong: A larger buffer does not change the format, and the CTAS query is the extra job the team does not want to run. The option misses both the format and the no-extra-job requirements.
Explanation
Firehose can convert records to Parquet or ORC during delivery, which avoids a separate job. It needs a Glue Data Catalog table that describes the schema, and it applies that schema to each record before writing to S3. Compression, a larger buffer, or a per-record function do not change the format in the same way.
A logistics broker's Athena tables sit on about 2 million small JSON files, each only a few kilobytes, written by an ingestion service every minute. Dashboards that filter by shipment date take minutes to run, and most of the time goes to opening files.
The team wants the data stored as partitioned, compressed Parquet files that are large enough to scan efficiently, and it wants the existing dashboards to keep working through the same table name pattern after a short cutover.
Which approach meets these requirements?
Choose one answer.
AA. Keep the JSON files, and add a daily Lambda function that gzips each small file in place so that the objects are smaller.
BB. Create one Athena table for each shipment day, and have the dashboards union the daily tables at query time.
CC. Increase the ingestion service's frequency so that each file holds more records, and keep the JSON format unchanged.
DD. Use an Athena CTAS query to write a new table in partitioned, compressed Parquet, then point the dashboards to that table after the cutover.
Show the answer and explanation
Answer: D
A. Wrong: Gzipping the small files does not reduce the number of objects, so the per-file overhead stays the same. The data also remains row-based JSON, which the team wants to move away from.
B. Wrong: One table per day multiplies the metadata to manage and makes every dashboard query more complex. The small files would still exist, so the scan cost stays high.
C. Wrong: Writing more often makes the files smaller, not larger. The option works against the goal and keeps the row-based format.
D. Right: Correct. CTAS reads the JSON once and writes compressed Parquet files partitioned by shipment date, with fewer and larger objects. Dashboards then query the new table, which lets Athena skip most partitions and read fewer files. Learn more: Create a table from query results (CTAS) (opens in a new tab)
Explanation
Many small files slow Athena because each file has overhead to open and read. A CTAS query rewrites the data once into fewer, larger, compressed Parquet files, partitioned by the column dashboards filter on. Gzipping small files, per-day tables and writing more often each leave the file count or format problem in place.
Want more on File formats and conversion? The 5 full-length practice sets have 9 more File formats and conversion questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →