Course menuDEA-C01

Exams / DEA-C01 / Study guide / Lambda for data processing

Lambda for data processing: DEA-C01 study notes and practice questions

What to know about Lambda for data processing for the DEA-C01 exam, from the Data Ingestion and Transformation domain (34% of the exam guide), then 1 free question to test it. Part of the DEA-C01 study guide.

Last verified against current AWS documentation ·How we write our questions

What to know for the exam

Lambda questions ask whether a data task fits Lambda, and how to make stream processing reliable.

Limits that decide questions

  • 15 minutes maximum run time.
  • 10,240 MB maximum memory (CPU scales with memory).
  • /tmp storage configurable up to 10 GB.
  • Payload limits for synchronous invocations (6 MB) and asynchronous ones.

Big or long transformations go to Glue, EMR or Batch.

Event source mappings

Lambda polls the source and invokes your function with batches:

Source Notes
Kinesis / DynamoDB Streams In order per shard; a failed batch blocks the shard until it succeeds or expires
SQS Failed messages return to the queue; use a dead-letter queue on the queue and partial batch responses
MSK / self-managed Kafka Polls topics

Making stream processing reliable

  • Bisect batch on function error splits a failing batch to isolate the bad record.
  • Maximum retry attempts and maximum record age stop endless retries.
  • On-failure destination (SQS, SNS, S3) keeps details of discarded batches.
  • Report batch item failures to retry only failed records.
  • Parallelization factor increases concurrency per shard.
  • Watch IteratorAge to see if the function keeps up.

Other data uses

  • Firehose data transformation: a Lambda function transforms each buffered batch before delivery.
  • S3 Object Lambda transforms data as it’s read.
  • Layers package shared libraries (such as pandas or AWS SDK for pandas).

Common traps

  • One bad record stalling a Kinesis shard for days: set retry limits and bisect.
  • Lambda for multi-hour ETL.
  • Unbounded concurrency overwhelming a database: set reserved concurrency or use RDS Proxy.

Try the 1 practice question

A clinic network's Lambda function reads a Kinesis data stream in batches of 500 records. One malformed record causes the function to fail, and Lambda retries the whole batch. The shard stalls for hours, and the records behind the bad record are not processed.

The team needs the good records to keep flowing, the malformed record to be kept for investigation, and no data to be lost without a trace. The stream and the function code should stay as they are.

Which change meets these requirements?

Choose one answer.

  1. A. Increase the function's timeout and memory, so that the function can process the malformed record before the batch is retried.
  2. B. Set the maximum retry attempts to unlimited, so that the function keeps retrying the batch until the malformed record succeeds.
  3. C. Turn on bisect on error, cap retry attempts and record age, and send discarded batch details to an on-failure queue.
  4. D. Move the stream's records to an Amazon SQS queue first, and have the function read from the queue instead of the stream.
Show the answer and explanation

Answer: C

  • A. Wrong: A malformed record fails the same way no matter how much time or memory the function has. The timeout and memory changes do not stop the retries or keep the shard moving.
  • B. Wrong: An unlimited retry count keeps the shard blocked on a record that cannot succeed. The good records behind it still wait, and the stall gets longer.
  • C. Right: Correct. Bisecting splits a failed batch so that the bad record is isolated while the good records are processed. The retry limit and record age stop the stall, and the on-failure destination keeps the details of the discarded batch for investigation. Learn more: Using Lambda to process records from Amazon Kinesis Data Streams (opens in a new tab)
  • D. Wrong: A queue can hold the malformed record, but the function still fails on it and retries it. Moving the consumer also changes the stream design that the team wants to keep.

Explanation

A failed Kinesis batch blocks its shard until Lambda stops retrying it, so a single bad record can stall everything behind it. Bisecting the batch, limiting retries and record age, and sending the discarded batch details to an on-failure destination isolate the bad record and keep the stream moving. More time, more memory or unlimited retries leave the blockage in place.

Study

Want more on Lambda for data processing? The 5 full-length practice sets have 8 more Lambda for data processing questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets

Study Lambda for data processing

More Data Ingestion and Transformation topics