Exams / SAP-C03 / Study guide / Serverless data pipelines and durable workflows (Step Functions)
Serverless data pipelines and durable workflows (Step Functions): SAP-C03 study notes and practice questions
What to know about Serverless data pipelines and durable workflows (Step Functions) for the SAP-C03 exam, from the Architecture Design and Implementation Built for the Cloud domain (25% of the exam guide), then 1 free question to test it. Part of the SAP-C03 study guide.
SAP-C03 asks two kinds of questions here: how to build a data processing pipeline from managed, serverless parts, and how to run a multi-step business process that must survive failures, wait for people and never lose its place.
Serverless data pipelines
Ingest: Kinesis Data Streams for ordered real-time records; Amazon Data Firehose to batch and deliver to S3, Redshift or OpenSearch with optional Lambda transformation; S3 event notifications for file drops.
Transform: Lambda for small, fast per-record work (15-minute maximum); AWS Glue jobs (Spark or Python shell) for large ETL; Glue crawlers populate the Glue Data Catalog.
Query:Athena runs SQL on S3 in place. Store data as Parquet or ORC, partition it, and compress it so each query scans less data and costs less.
Orchestrate: Step Functions or EventBridge Scheduler runs the jobs in order.
Step Functions: Standard versus Express
Standard
Express
Duration
Up to 1 year
Up to 5 minutes
Semantics
Exactly-once workflow execution
At least once (asynchronous) or at most once (synchronous)
Pricing
Per state transition
Per request, duration and memory
History
Full visual execution history
CloudWatch Logs
Fits
Long, auditable business processes
High-volume, short event processing
Useful features: retry and catch per state with exponential backoff, the .waitForTaskToken callback pattern for human approvals or external systems, .sync to run a job (Glue, ECS, EMR) and wait for it, Map and Distributed Map for parallel processing of large S3 datasets, and direct SDK integrations with AWS services, so many steps need no Lambda function at all.
Lambda durable functions
Durable functions let you write a multi-step workflow as ordinary sequential code in a Lambda function. The SDK checkpoints each step; after a failure or pause, the function replays from the start and skips completed steps using stored results. A durable execution can run up to one year, and waits don’t incur compute charges, so workflows can pause for hours or days, including for human approval.
How to choose
Scenario wording
Choose
“Visual workflow across many AWS services, audit history”
Step Functions Standard
“Thousands of short executions per second, lowest cost”
Step Functions Express
“Developers want the workflow in application code in Lambda”
Lambda durable functions
“Wait days for a manager to approve”
.waitForTaskToken or a durable wait
“Ad hoc SQL on raw files in S3, no infrastructure”
Athena with the Glue Data Catalog
“Process millions of S3 objects in parallel”
Step Functions Distributed Map
Common traps
Chaining Lambda functions that call each other synchronously, so failures and timeouts cascade and you pay for idle waits.
Express workflows for a process that runs for hours or must not run twice.
Polling in a loop for an approval instead of a task token callback.
Querying uncompressed CSV with Athena when the ask is lower cost and faster queries.
A long-running EC2 scheduler to sequence Glue jobs.
A retailer runs product returns as a multi-step workflow. For each return, the workflow checks the order, issues a refund through a payment provider, books a pickup with a courier and waits for a store manager's approval when the refund is above a set amount. Courier pickup confirmations can take up to 30 days.
Each refund must happen at most once, even when a step is retried. Auditors need a complete record of every step for each return, and the operations team does not want custom code that stores workflow state or polls for approvals.
Which solution meets these requirements?
Choose one answer.
AA. Run each return as an AWS Step Functions Standard Workflow. Each task retries with backoff, the approval and courier steps pause on a task token, and the refund call uses an idempotency key built from the return ID.
BB. Run each return as an AWS Step Functions Express Workflow with retries on every task and Wait states for the approval and the courier confirmation. Send the execution logs to CloudWatch Logs for the audit trail.
CC. Store each return's state in an Amazon DynamoDB table, use Amazon EventBridge Scheduler to run a Lambda function every few minutes to check the courier and approval status, and call the refund API from that function.
DD. Pass each step through its own Amazon SQS queue, with a visibility timeout of 30 days on the courier confirmation queue, and have a consumer Lambda function for each queue call the payment provider or the courier.
Show the answer and explanation
Answer: A
A. Right: Correct. Standard workflows can run for long periods and wait for external callbacks with task tokens, so no polling code is needed. The idempotency key keeps a retried refund from being paid twice, and the execution history gives auditors the step-by-step record. Learn more: Call Step Functions with a task token (opens in a new tab)
B. Wrong: Express workflows are built for short, high-volume runs and cannot run longer than five minutes, so a 30-day wait cannot be modeled in one execution. Their at-least-once processing also makes duplicate refunds more likely.
C. Wrong: This design can work, but the team would write and run the state tracking, the polling and the retry logic itself. That is the custom code the operations team wants to avoid, and the audit record would also need building.
D. Wrong: SQS limits the visibility timeout to 12 hours, so a 30-day confirmation cannot be held this way. The functions would also need their own protection against duplicate refunds, and queue messages do not give auditors a complete history of each return.
Explanation
Long waits for human approval and courier callbacks, at-most-once refunds and a full history without custom state code point to a Standard workflow. Express workflows have a short maximum duration, and polling or queue-based designs add custom code or hit service limits.
Want more on Serverless data pipelines and durable workflows (Step Functions)? The 5 full-length practice sets have 5 more Serverless data pipelines and durable workflows (Step Functions) questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →
Study Serverless data pipelines and durable workflows (Step Functions)