Orchestration with Step Functions, MWAA and Glue workflows: DEA-C01 study notes and practice questions
What to know about Orchestration with Step Functions, MWAA and Glue workflows for the DEA-C01 exam, from the Data Ingestion and Transformation domain (34% of the exam guide), then 2 free questions to test it. Part of the DEA-C01 study guide.
Orchestration questions ask how to run several steps in order, handle failures, and see what happened.
The options
Service
Best for
Notes
AWS Step Functions
AWS-native workflows across many services
Visual state machines; direct service integrations (Glue, EMR, Athena, Lambda, Batch, ECS, SNS…) with .sync to wait for jobs; retries, catch, parallel and map states
Amazon MWAA
Teams using Apache Airflow and Python DAGs
Managed Airflow; DAGs in S3; many operators; good for complex, cross-system pipelines
AWS Glue workflows
Pipelines made only of Glue jobs and crawlers
Triggers chain jobs and crawlers; simple and built in
EventBridge
Starting pipelines on events or schedules
Not a full orchestrator
Step Functions types
Standard
Express
Duration
Up to 1 year
Up to 5 minutes
Execution
Exactly once
At least once
Price
Per state transition
Per request and duration
Fits
Long ETL with Glue or EMR jobs, human approvals
High-volume, short event processing
Error handling
Retry with backoff for transient errors; Catch to route failures to a notification or cleanup step.
Distributed Map processes millions of S3 objects in parallel.
In Airflow, task retries, SLAs and callbacks.
Troubleshooting MWAA
A DAG that doesn’t appear usually failed to parse: check Import Errors in the Airflow UI and the scheduler logs in CloudWatch.
Missing Python packages go in requirements.txt; check the environment update logs.
The execution role needs access to S3, CloudWatch Logs and any services the DAG calls.
Common traps
Express workflows for a 2-hour Glue job. Use Standard.
Chaining Lambda functions to wait for long jobs. Use Step Functions .sync integrations.
MWAA for a two-job Glue pipeline. Glue workflows or Step Functions are simpler.
A logistics broker's nightly pipeline runs a Glue job that takes about two hours, followed by a Glue job that rebuilds the reporting tables. The pipeline must start each step only after the previous one finishes successfully, retry transient failures, and notify the on-call team if a step fails.
Each step writes to a shared table, so a duplicate execution would corrupt the data. The team wants AWS to manage the state of the run.
Which orchestration approach meets these requirements?
Choose one answer.
AA. Use an AWS Step Functions Express workflow, which starts quickly and keeps each run's state in the workflow.
BB. Use a Step Functions Standard workflow with a Glue task that waits for each job, retry rules, and a catch path that publishes to SNS.
CC. Use an Amazon EventBridge rule that starts the second Glue job when the first Glue job's state changes, and an SNS topic for alerts.
DD. Use an AWS Lambda function on a schedule that polls the Glue job status every minute and starts the next job when the first one is done.
C. Wrong: EventBridge can react to a Glue job state change, but a rule does not track the run as a whole or retry a failed step. The team would have to build the sequencing and retry logic itself, which leaves AWS managing less of the run's state.
Step Functions Standard workflows run for up to a year and execute each step exactly once, which suits a long Glue job that writes shared tables. Waiting tasks, retry rules and catch paths give the sequencing, retries and alerts the team needs, and AWS keeps the run's state. Express workflows, EventBridge rules and polling functions each miss one of these.
A data team runs its pipelines in an Amazon Managed Workflows for Apache Airflow environment. After the team uploads a new DAG file to the environment's dags folder in S3, the DAG never appears in the Airflow UI, and no DAG run is scheduled. The DAG imports Glue and Redshift operators from provider packages.
The team needs to find the cause without deleting or recreating the environment.
Which TWO steps help identify the cause? (Choose two.)
Choose two answers.
AA. Delete the environment and create a new one with the same DAG folder, so that Airflow starts with a clean state.
BB. Move the DAG's logic into an AWS Lambda function and trigger that function from an EventBridge rule, so that Airflow no longer needs to load the file.
CC. Check the Import Errors list in the Airflow UI and the scheduler logs in CloudWatch for parse errors raised when the DAG file is loaded.
DD. Update the environment to restart the Airflow components, and wait for the scheduler to rescan the DAG folder after the update.
EE. Confirm that every provider package the DAG imports is listed in requirements.txt, since a missing package stops parsing.
Show the answer and explanation
Answers: C, E
A. Wrong: Recreating the environment is slow and disruptive, and the new environment would have the same DAG file and the same problem. The team asked to avoid this step.
B. Wrong: Moving the logic out of Airflow replaces the pipeline design rather than diagnosing it. The team wants the existing DAG to run in MWAA, so this option does not answer the question.
C. Right: Correct. Airflow lists DAG files that fail to parse as import errors in its UI, and MWAA can publish DAG processing and scheduler logs to CloudWatch Logs. A syntax error or a failed import shows up there and points to the cause directly. Learn more: Accessing Airflow logs in Amazon CloudWatch (opens in a new tab)
D. Wrong: Restarting the components does not fix a DAG file that fails to parse, so the scheduler would fail the same way again. The option also takes the environment through an update without telling the team what is wrong.
E. Right: Correct. Packages that the DAG imports must be installed in the environment, and they are declared in requirements.txt. A missing or incompatible package causes an import error, which keeps the DAG from loading. Learn more: Installing Python dependencies (opens in a new tab)
Explanation
A DAG that does not appear usually failed to parse. The Airflow Import Errors view and the scheduler logs in CloudWatch show parse errors, and missing provider packages in requirements.txt are a common cause. Recreating the environment, moving the logic out of Airflow or restarting the web server does not diagnose the problem.
Want more on Orchestration with Step Functions, MWAA and Glue workflows? The 5 full-length practice sets have 11 more Orchestration with Step Functions, MWAA and Glue workflows questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →
Study Orchestration with Step Functions, MWAA and Glue workflows