ExamPark
Course menuDEA-C01

Exams / DEA-C01 / Study guide / Troubleshooting transformations

Troubleshooting transformations: DEA-C01 study notes and practice questions

What to know about Troubleshooting transformations for the DEA-C01 exam, from the Data Ingestion and Transformation domain (34% of the exam guide), then 1 free question to test it. Part of the DEA-C01 study guide.

Last verified against current AWS documentation ·How we write our questions

What to know for the exam

These questions describe a slow or failing Spark or Glue job and ask for the cause or the fix.

Common problems

Symptom Likely cause Fix
One task runs far longer than the rest; others finish quickly Data skew: a key (often null or a default value) holds a large share of rows Filter or handle nulls separately, salt the hot key, broadcast the small side of a join, use adaptive query execution
One task per huge file, out of memory Unsplittable compression (one big gzip file) Split files, or convert to Parquet
Executors fail with out-of-memory Too little memory per executor, large shuffles, collecting data to the driver Larger worker type, more partitions, avoid collect()
Job slow with thousands of tiny files Per-file overhead Group files on read (groupFiles), compact output
Write fails on type errors Mixed or changed column types resolveChoice, explicit casting, schema checks
Job reprocesses old data Job bookmarks off or reset Turn bookmarks on and keep transformation contexts stable
Can’t reach a JDBC source Glue connection: subnet, security group, route to the database, endpoint for S3 Fix the connection’s networking

Tools

  • Spark UI (Glue and EMR) shows stages, tasks and their durations: the way to see skew.
  • Glue job metrics and Glue job run insights in CloudWatch: memory, data shuffled, executor activity, and likely root causes.
  • CloudWatch Logs for driver and executor logs.
  • Glue observability metrics for job and resource health.

Prevention

  • Profile data first (DataBrew or a small sample job).
  • Validate input with Glue Data Quality rules.
  • Test on representative data volumes, not tiny samples.

Common traps

  • Adding workers to fix skew. The one heavy task still runs alone.
  • Blaming the cluster size for a gzip file that can’t be split.

Try the 1 practice question

Practice Troubleshooting transformationsTake the free practice exam

Question 1

Harder ·Link to this question (question 1)

A property-management company's nightly AWS Glue Spark job reads 40 gzip-compressed CSV files, and each file is about 2 GB. Several executors fail with out-of-memory errors, and the Spark UI shows a few tasks running for a long time while most of the workers sit idle.

The company wants the job to finish within its nightly window without losing rows, and it wants to avoid a large cost increase.

Which change meets these requirements?

Choose one answer.

  1. A. Split the source into moderately sized files, or convert it to a splittable format such as Parquet, so Spark can run tasks in parallel.
  2. B. Change every worker to G.8X and double the number of workers, so that each executor has more memory to hold the decompressed file.
  3. C. Reduce the job's parallelism by setting the number of shuffle partitions to 1, so that only one task runs at a time and uses less memory.
  4. D. Add a Glue crawler step that runs before the job, so that the catalog holds the file sizes and Spark can plan the reads better.
Show the answer and explanation

Answer: A

  • A. Right: Correct. A gzip file cannot be split, so Spark reads each 2 GB file in a single task, and a few huge partitions run out of executor memory while other workers wait. Smaller files or a splittable format let Spark spread the work over many smaller tasks. Learn more: Monitoring jobs using the Apache Spark web UI (opens in a new tab)
  • B. Wrong: Bigger workers would raise the cost sharply, and a single gzip file still runs in one task. The idle workers would stay idle, so the option does not address the cause.
  • C. Wrong: One task at a time makes the job slower and does not fix the big-file problem. The option works against the nightly window.
  • D. Wrong: A crawler records schema and location metadata, and it does not change how Spark splits the file. The option leaves the gzip problem in place.

Explanation

Gzip is not splittable, so Spark assigns each large gzip file to one task, which can run out of memory while the rest of the cluster waits. Splitting the data into moderately sized files or converting it to a splittable format such as Parquet lets the same workers finish the job in the window. Larger workers and fewer tasks raise cost or slow the job.

Study

Want more on Troubleshooting transformations? The 5 full-length practice sets have 7 more Troubleshooting transformations questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets

Study Troubleshooting transformations

More Data Ingestion and Transformation topics