DEA-C01 practice question: Troubleshooting transformations

Data Ingestion and Transformation A free question from our original DEA-C01 practice exam. Try it, then open the answer to see why each option is right or wrong.

A property-management company's nightly AWS Glue Spark job reads 40 gzip-compressed CSV files, and each file is about 2 GB. Several executors fail with out-of-memory errors, and the Spark UI shows a few tasks running for a long time while most of the workers sit idle.

The company wants the job to finish within its nightly window without losing rows, and it wants to avoid a large cost increase.

Which change meets these requirements?

Choose one answer.

A. Split the source into moderately sized files, or convert it to a splittable format such as Parquet, so Spark can run tasks in parallel.
B. Change every worker to G.8X and double the number of workers, so that each executor has more memory to hold the decompressed file.
C. Reduce the job's parallelism by setting the number of shuffle partitions to 1, so that only one task runs at a time and uses less memory.
D. Add a Glue crawler step that runs before the job, so that the catalog holds the file sizes and Spark can plan the reads better.

Show the answer and explanation

Answer: A

A. Right: Correct. A gzip file cannot be split, so Spark reads each 2 GB file in a single task, and a few huge partitions run out of executor memory while other workers wait. Smaller files or a splittable format let Spark spread the work over many smaller tasks. Learn more: Monitoring jobs using the Apache Spark web UI (opens in a new tab)
B. Wrong: Bigger workers would raise the cost sharply, and a single gzip file still runs in one task. The idle workers would stay idle, so the option does not address the cause.
C. Wrong: One task at a time makes the job slower and does not fix the big-file problem. The option works against the nightly window.
D. Wrong: A crawler records schema and location metadata, and it does not change how Spark splits the file. The option leaves the gzip problem in place.

Explanation

Gzip is not splittable, so Spark assigns each large gzip file to one task, which can run out of memory while the rest of the cluster waits. Splitting the data into moderately sized files or converting it to a splittable format such as Parquet lets the same workers finish the job in the window. Larger workers and fewer tasks raise cost or slow the job.

DEA-C01 practice question: Troubleshooting transformations

Explanation

Study

Keep practicing