ExamPark
Course menuDEA-C01

Exams / DEA-C01 / Study guide / Troubleshooting and tuning pipelines

Troubleshooting and tuning pipelines: DEA-C01 study notes and practice questions

What to know about Troubleshooting and tuning pipelines for the DEA-C01 exam, from the Data Operations and Support domain (22% of the exam guide), then 3 free questions to test it. Part of the DEA-C01 study guide.

Last verified against current AWS documentation ·How we write our questions

What to know for the exam

These questions give a symptom and ask for the most likely cause or the fix with least effort. Match the symptom to the cause.

Symptom to cause

Symptom Likely cause Fix
Kinesis ProvisionedThroughputExceeded Hot shard or too few shards Better partition key, more shards or on-demand mode
Rising iterator age Consumers too slow More parallelism (Lambda parallelization factor), enhanced fan-out, more shards
Glue job out of memory on the driver Collecting data to the driver or too many small files listed Avoid collect(), use groupFiles, larger workers
One Spark task runs far longer Data skew on a join or group key Salt the key, broadcast the small table, repartition
Athena query slow and costly CSV, no partitions, small files Parquet, partitions, compaction
New data missing in Athena Partitions not registered Crawler, MSCK REPAIR TABLE, partition projection
Redshift one slice busy Distribution skew Change the distribution key
AccessDenied writing to S3 Role policy, bucket policy or KMS key policy Grant kms:GenerateDataKey/kms:Decrypt as well as S3 actions
Glue job can’t reach a database VPC connection, security group, missing self-referencing rule or no route to S3 Fix the Glue connection and add an S3 gateway endpoint
Lambda times out on batches Batch too large, slow downstream Smaller batch, more concurrency, longer timeout up to 15 minutes
Duplicate records downstream At-least-once delivery and retries Idempotent writes, dedupe on a key
Glue job reprocesses old files Job bookmarks disabled or reset Enable bookmarks

Approach

  1. Check the job’s logs and error messages (CloudWatch Logs, Spark UI, Step Functions history).
  2. Check metrics for throttling, lag or skew.
  3. Check permissions: IAM role, resource policy, KMS key policy, Lake Formation grants.
  4. Check network paths for VPC resources.

Common traps

  • Scaling up when the cause is skew; one hot key stays hot.
  • Fixing only the S3 policy when the bucket uses a customer managed KMS key.
  • Retrying forever without a dead-letter queue.

Try the 3 practice questions

Practice Troubleshooting and tuning pipelinesTake the free practice exam

Question 1

Medium ·Link to this question (question 1)

A bank's nightly AWS Glue job joins 900 million card transactions with a merchants table. The job's Spark UI shows that 199 of the 200 tasks in the join stage finish within a minute, while one task runs for almost two hours and holds most of the shuffle data.

Investigation shows that about 60% of the transactions have a null merchant_id, and all of those rows hash to the same join key. The team wants the job to finish inside its window without changing the output.

Which change meets these requirements?

Choose one answer.

  1. A. Double the number of workers, so that the slow task has more memory and cores to finish sooner.
  2. B. Change the output format from Parquet to CSV, so that the write stage finishes faster and the join stage has less work.
  3. C. Set the shuffle partitions to 1, so that the whole join runs in one task and avoids the network transfer of the shuffle data.
  4. D. Split the rows with a null merchant_id out before the join, then union them back afterward with empty merchant columns.
Show the answer and explanation

Answer: D

  • A. Wrong: One task holds all the null-key rows, and a single task runs on a single core regardless of how many workers the job has. More workers do not split that task, so the job still waits on it.
  • B. Wrong: The join stage is the slow part, and the output format does not change the join. The option does not address the skewed key.
  • C. Wrong: One partition makes the single task carry all the data, which makes the skew worse. The option works against the goal of finishing in the window.
  • D. Right: Correct. Null keys never match in the join, yet they all land in one shuffle partition, which creates the hot task. Joining only the rows that have a merchant_id and adding the null rows back with empty merchant columns gives the same output as the original left join. Learn more: Monitoring jobs using the Apache Spark web UI (opens in a new tab)

Explanation

The Spark UI shows one task doing most of the work, which is data skew. Here a null key puts 60% of the rows in one join partition even though those rows cannot match. Splitting the null rows out and adding them back after the join spreads the remaining work across tasks and leaves the output unchanged. Adding workers, changing formats or shrinking parallelism does not split the hot key.

Study

Question 2

Medium ·Link to this question (question 2)

A dental chain's Athena queries on a claims table began failing last Tuesday with a HIVE_PARTITION_SCHEMA_MISMATCH error after the upstream system changed one column's type from int to bigint. Only the partitions written before the change are affected, and the table's own schema already shows the new type.

The team wants the queries to work again on all partitions without rewriting the data files.

Which action should the team take?

Choose one answer.

  1. A. Drop the affected partitions and add them again, for example with MSCK REPAIR TABLE, so that they take the table's current schema.
  2. B. Run MSCK REPAIR TABLE on the claims table, which updates the column types of every existing partition to match the table.
  3. C. Delete the claims table and recreate it with the new column type, then run a crawler to find all of the partitions again.
  4. D. Rewrite all of the old data files in the new type with a CTAS query, and point the table at the new files.
Show the answer and explanation

Answer: A

  • A. Right: Correct. Athena stores a schema for the table and for each partition, and the error means the old partitions still carry the int type. Dropping those partitions and re-adding them registers them with the table's schema, and the data files stay where they are. Learn more: Troubleshoot issues in Athena (opens in a new tab)
  • B. Wrong: MSCK REPAIR TABLE adds partitions that exist in S3 but are missing from the catalog. It does not change the column types of partitions that already exist, so the mismatch remains. Learn more: MSCK REPAIR TABLE (opens in a new tab)
  • C. Wrong: Recreating the table can work, but it removes the table definition and its settings, and it is more work than the fix needs. The option is not the minimal change the team wants.
  • D. Wrong: A rewrite changes the data, which the team wants to avoid, and the cost of rewriting a full claims table is high. The option is more work than updating partition metadata.

Explanation

Athena checks each partition's column types against the table schema, and partitions registered before the type change trigger the mismatch error. Dropping and re-adding those partitions, for example with MSCK REPAIR TABLE, gives them the table's schema without touching the data. Running MSCK REPAIR alone does not change partitions that already exist, and rewriting or recreating the table does more than needed.

Study

Question 3

Harder ·Link to this question (question 3)

A cooperative grocery group's nightly Amazon Redshift load starts on time, but its final step waits for more than an hour on a table lock. The team's monitoring shows that the queries are queued, not running. An analyst's session ran an UPDATE on the same table earlier that evening, and the session is still open with no statement running.

The team needs the nightly load to finish tonight.

Which action should the team take tonight to unblock the load?

Choose one answer.

  1. A. Add more WLM queue slots to the nightly load's queue, so that the waiting step can run alongside the blocked one.
  2. B. Query SVV_TRANSACTIONS to find the open transaction holding the lock, then end that idle session so the nightly load can run.
  3. C. Run VACUUM FULL on the table, which rewrites the rows and releases the lock held by the open session.
  4. D. Enable concurrency scaling on the cluster, so that the queued load runs on a separate cluster that does not share the table lock.
Show the answer and explanation

Answer: B

  • A. Wrong: Queue slots control how many queries run at once, but a lock still blocks the step that needs the table. More slots do not release the lock held by the open session.
  • B. Right: Correct. SVV_TRANSACTIONS shows which transactions currently hold locks, so the team can find the idle session. Ending that session releases the lock, and the load runs. Applications should commit or roll back promptly to avoid the same wait. Learn more: SVV_TRANSACTIONS (opens in a new tab)
  • C. Wrong: VACUUM FULL needs the same table lock and waits behind the open transaction. It does not release a lock that another session holds.
  • D. Wrong: Concurrency scaling adds capacity for queued queries, but the lock belongs to a transaction on the main cluster. The load would still wait until that transaction ends.

Explanation

A queued query on a locked table is waiting for a transaction, not for capacity. SVV_TRANSACTIONS shows the open transaction that holds the lock, and ending or finishing that session releases the table so the load can run. Queue slots, VACUUM FULL and concurrency scaling do not release a lock held by another session.

Study

Want more on Troubleshooting and tuning pipelines? The 5 full-length practice sets have 13 more Troubleshooting and tuning pipelines questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets

Study Troubleshooting and tuning pipelines

More Data Operations and Support topics