DEA-C01 practice question: Troubleshooting and tuning pipelines (1 of 3)
Data Operations and Support A free question from our original DEA-C01 practice exam. Try it, then open the answer to see why each option is right or wrong.
A bank's nightly AWS Glue job joins 900 million card transactions with a merchants table. The job's Spark UI shows that 199 of the 200 tasks in the join stage finish within a minute, while one task runs for almost two hours and holds most of the shuffle data.
Investigation shows that about 60% of the transactions have a null merchant_id, and all of those rows hash to the same join key. The team wants the job to finish inside its window without changing the output.
Which change meets these requirements?
Choose one answer.
AA. Double the number of workers, so that the slow task has more memory and cores to finish sooner.
BB. Change the output format from Parquet to CSV, so that the write stage finishes faster and the join stage has less work.
CC. Set the shuffle partitions to 1, so that the whole join runs in one task and avoids the network transfer of the shuffle data.
DD. Split the rows with a null merchant_id out before the join, then union them back afterward with empty merchant columns.
Show the answer and explanation
Answer: D
A. Wrong: One task holds all the null-key rows, and a single task runs on a single core regardless of how many workers the job has. More workers do not split that task, so the job still waits on it.
B. Wrong: The join stage is the slow part, and the output format does not change the join. The option does not address the skewed key.
C. Wrong: One partition makes the single task carry all the data, which makes the skew worse. The option works against the goal of finishing in the window.
D. Right: Correct. Null keys never match in the join, yet they all land in one shuffle partition, which creates the hot task. Joining only the rows that have a merchant_id and adding the null rows back with empty merchant columns gives the same output as the original left join. Learn more: Monitoring jobs using the Apache Spark web UI (opens in a new tab)
Explanation
The Spark UI shows one task doing most of the work, which is data skew. Here a null key puts 60% of the rows in one join partition even though those rows cannot match. Splitting the null rows out and adding them back after the join spreads the remaining work across tasks and leaves the output unchanged. Adding workers, changing formats or shrinking parallelism does not split the hot key.
Want more on Troubleshooting and tuning pipelines? The 5 full-length practice sets have 13 more Troubleshooting and tuning pipelines questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →