Monitoring and logging pipelines: DEA-C01 study notes and practice questions
What to know about Monitoring and logging pipelines for the DEA-C01 exam, from the Data Operations and Support domain (22% of the exam guide), then 3 free questions to test it. Part of the DEA-C01 study guide.
Monitoring questions ask how to find out quickly that a pipeline is failing, slow or falling behind, ideally automatically.
What to watch
Service
Key signals
Kinesis Data Streams
GetRecords.IteratorAgeMilliseconds (consumers falling behind), WriteProvisionedThroughputExceeded and ReadProvisionedThroughputExceeded
Firehose
Delivery success and data freshness; failed records go to an S3 error prefix
Glue jobs
Job run state, duration, DPU usage, Spark UI and job run insights, continuous logging to CloudWatch Logs
Lambda
Errors, throttles, duration, iterator age for stream sources
Redshift
CPU, disk usage, query duration, WLM queue wait; system tables and views such as SYS_QUERY_HISTORY
MWAA / Step Functions
Task failures and execution history
DMS
CDC latency (source and target)
Tools
CloudWatch metrics and alarms with SNS notifications; anomaly detection for metrics without a fixed threshold.
CloudWatch Logs and Logs Insights to search and aggregate logs with a query language; metric filters turn log patterns into metrics.
EventBridge rules react to state changes such as a Glue job FAILED, a crawler finishing or a Step Functions execution failing, and can notify or start a retry.
CloudTrail records API calls: who changed a job, deleted a table or altered a bucket policy. CloudTrail Lake lets you query events with SQL.
AWS Config tracks resource configuration over time.
Amazon Managed Grafana and Quick Sight for dashboards.
Designing alerts
Alert on symptoms users feel: data freshness, failed runs, row counts outside the usual range.
Send failures to a team channel through SNS or Amazon Q Developer in chat applications.
Use dead-letter queues or error prefixes so bad records are kept for review.
Common traps
CloudTrail for performance metrics. It records API calls; CloudWatch holds metrics.
Polling job status in a script when an EventBridge rule can react to state changes.
A ticketing company's Amazon Redshift cluster uses manual WLM, with one queue for ETL and one for analysts. Some ad hoc analyst queries run for hours and scan billions of rows, and the nightly ETL slows down while they run.
The data team wants long-running analyst queries recorded for review, and any analyst query that scans more than 5 billion rows stopped automatically. ETL queries must not be affected, and the team does not want to write scripts.
Which solution meets these requirements?
Choose one answer.
AA. Schedule a Lambda function every five minutes that queries STV_RECENTS and cancels any analyst query that has run for more than one hour.
BB. Add query monitoring rules to the analyst queue, with a log action for long run times and an abort action for queries scanning over 5 billion rows.
CC. Set the statement_timeout parameter in the cluster's parameter group, so that every query stops after it has run for one hour.
DD. Turn on concurrency scaling for the analyst queue, so that the long analyst queries run on extra capacity and leave the ETL queue alone.
Show the answer and explanation
Answer: B
A. Wrong: This is the script the team wants to avoid, and it judges queries by run time rather than by the rows they scan. A large scan could also do its damage between two polls.
B. Right: Correct. Query monitoring rules belong to one WLM queue, so ETL queries are not affected. A log action records the query in STL_WLM_RULE_ACTION, and an abort action on the scan_row_count metric cancels the large scans without any script. Learn more: WLM query monitoring rules (opens in a new tab)
C. Wrong: A cluster-wide timeout also stops long ETL statements, which must not be affected. It limits run time only, so a short query that scans billions of rows still runs.
D. Wrong: Concurrency scaling adds capacity for queued queries, but it neither records nor stops the expensive ones. The large scans still run and add cost.
Explanation
Redshift query monitoring rules watch the queries in one WLM queue and act when their metrics cross a threshold. A log action keeps a record for review, and an abort action on rows scanned stops the expensive queries, with no effect on other queues. Polling scripts, cluster-wide timeouts and extra capacity miss the scope, the metric or the no-script requirement.
A property-management company runs 14 AWS Lambda functions that make up its rent-ingestion pipeline. Each function writes its logs to its own CloudWatch log group. After a bad week, the operations lead wants to see, for the last seven days, how many errors each function raised and which error types were most common.
The analysis is a one-time investigation, and the team wants it done in the console without building a new data store.
Which approach meets these requirements?
Choose one answer.
AA. Create an OpenSearch Service domain, subscribe every log group to it with a subscription filter, and build dashboards for the error counts.
BB. Create export tasks that copy the 14 log groups to S3, define Athena tables over the exported files, and query them to count the errors.
CC. Run one CloudWatch Logs Insights query over the 14 log groups for seven days, filtering errors and counting them by function and type.
DD. Turn on AWS X-Ray tracing for the 14 functions, and read each trace to count the errors across the seven days.
Show the answer and explanation
Answer: C
A. Wrong: OpenSearch works for ongoing search, but building a domain and subscriptions for a one-time investigation is more work and cost than the team needs. The option also creates a new data store, which the team wants to avoid.
B. Wrong: Exports can cover the past seven days, but each log group needs its own export task, and the team must then define tables over the files. The option adds a new data store and several steps that a one-time investigation does not need.
D. Wrong: X-Ray traces requests across services and does not give a log-wide count of error types over the week. Reading traces one by one is slow for this question.
Explanation
CloudWatch Logs Insights queries several log groups in a single query over a time range, and its aggregation commands count results by field. That answers the one-time question in the console without new infrastructure. OpenSearch, Athena exports and X-Ray take more setup or answer a different question.
A retail bank has a Kinesis Data Streams stream that feeds a Lambda function, which writes transactions to an analytics store. The team has seen several incidents in which the function fell behind during busy hours, and records sat in the stream for many minutes before anyone noticed.
The team wants an alarm that fires as soon as the consumer falls behind, using metrics that AWS already publishes, with no change to the function's code.
Which TWO actions meet these requirements? (Choose two.)
Choose two answers.
AA. Enable AWS CloudTrail data events for the stream, and query the events in CloudTrail Lake for each slow record.
BB. Increase the Lambda function's memory setting and its timeout, so that the function processes each batch faster and never falls behind.
CC. Create a CloudWatch alarm on the stream's GetRecords.IteratorAgeMilliseconds metric, which shows how long records waited.
DD. Create a CloudWatch alarm on the Lambda function's IteratorAge metric, which Lambda publishes for stream sources.
EE. Add a CloudWatch Logs subscription filter that sends the function's logs to a Lambda function that counts the log lines each minute.
Show the answer and explanation
Answers: C, D
A. Wrong: CloudTrail data events record API calls, not how far behind a consumer is. Querying them per record is slow and does not alarm when the lag grows.
B. Wrong: More memory can speed up processing, but the team wants an alarm that reports lag, not a faster function. The option does not add monitoring and cannot promise no lag under peak load.
E. Wrong: Counting log lines measures output volume, not how old the records are. The option does not describe the lag and would not alarm on it.
Explanation
Iterator age is the measure of consumer lag. Kinesis publishes an iterator age metric for the stream, and Lambda publishes one for functions that read from a stream, so an alarm on either shows when records wait too long. CloudTrail, memory tuning and log counts do not report how far behind the consumer is.
Want more on Monitoring and logging pipelines? The 5 full-length practice sets have 12 more Monitoring and logging pipelines questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →