Throttling, fan-out and replay: DEA-C01 study notes and practice questions
What to know about Throttling, fan-out and replay for the DEA-C01 exam, from the Data Ingestion and Transformation domain (34% of the exam guide), then 2 free questions to test it. Part of the DEA-C01 study guide.
These questions describe consumers falling behind, producers getting throttled, or a need to reprocess old data.
Consumer throughput
Standard (shared) consumers split each shard’s 2 MB/s read limit and 5 GetRecords calls per second. Each extra consumer gets less.
Enhanced fan-out gives each registered consumer its own 2 MB/s per shard pushed over HTTP/2, with lower latency. Use it for several consumers that each need full speed.
The IteratorAge (GetRecords.IteratorAgeMilliseconds) metric shows how far behind a consumer is.
Producer throttling
ProvisionedThroughputExceededException means a shard’s write limit was hit. Fixes: a better partition key (if one shard is hot), more shards (resharding: split hot shards) or on-demand mode, and retries with exponential backoff.
The Kinesis Producer Library (KPL) batches and aggregates records for efficiency.
Replay
You can only replay what the stream still holds. Retention defaults to 24 hours and can be raised up to 365 days (with extra cost beyond 24 hours, and long-term retention beyond 7 days).
Replay by starting a consumer at a timestamp or sequence number.
For longer history, land the raw stream in S3 with Firehose and reprocess from there.
MSK keeps data per topic retention and lets consumers reset offsets.
Lambda consumers
Parallelization factor (up to 10) processes several batches per shard at once, keeping order per key.
Batch size and batching window trade latency for efficiency.
Failed batches block the shard; use bisect batch on error, maximum retry attempts, maximum record age and an on-failure destination.
Common traps
Adding consumers without enhanced fan-out and slowing everyone down.
Planning a 3-day replay with default 24-hour retention.
A regional bank streams card authorization events into a Kinesis Data Streams stream with 12 shards. Four independent consumers read the full stream: fraud scoring, a risk dashboard, a data lake writer and an audit service.
With standard consumers, all of them share the read throughput of each shard, so the fraud consumer falls behind at peak times. The bank has already sized the stream and does not want to add shards, and every consumer must keep receiving records with low latency.
Which change meets these requirements?
Choose one answer.
AA. Double the shard count to 24 so that the shared read throughput of each shard is spread across more capacity.
BB. Have the fraud consumer increase its GetRecords batch size and poll more often, so that each request returns more records.
CC. Register each consumer as an enhanced fan-out consumer, which gives each one dedicated read throughput on every shard.
DD. Write each event twice from the producer, once to the stream for fraud scoring and once to a second stream for the other three consumers.
Show the answer and explanation
Answer: C
A. Wrong: More shards add read capacity, but the bank has ruled out changing the shard count. Consumers would still share read throughput on each shard, and the cost of the stream would rise.
B. Wrong: Larger polling batches do not raise the per-shard read limit that the consumers share. The standard consumers would still compete for the same read throughput on each shard.
D. Wrong: A second stream removes the sharing, but it doubles the producer work and cost and creates two copies that can drift apart. The bank wants a single stream as the source of record.
Explanation
Standard Kinesis consumers share each shard's read throughput, so more consumers mean less for each. Enhanced fan-out gives each registered consumer its own read throughput on every shard, which fixes the lag without changing the shard count or duplicating the stream. Bigger batches and extra copies of the stream do not change the shared limit.
A pet insurer's claim-events stream in Kinesis Data Streams feeds a fraud model and a reporting job. Last week a bug in the reporting job forced a replay, and the team found that records older than one day were already gone.
The stream still uses the default retention period. From now on, the team must be able to reprocess up to three days of events with the same consumer code, keeping the stream and the consumers as they are.
Which change meets this requirement?
Choose one answer.
AA. Increase the number of shards so that the stream can hold more records before the oldest ones expire.
BB. Set the reporting consumer's checkpoint back to the start of the stream, because the stream keeps records until they have been read.
CC. Turn on enhanced fan-out for the reporting consumer, so it can read older records from each shard at a higher rate and with its own throughput.
DD. Extend the stream's retention to at least three days, so that a consumer can later restart from a timestamp in that window.
Show the answer and explanation
Answer: D
A. Wrong: Shard count controls throughput, not how long records are kept. Adding shards leaves the default retention in place, so the oldest events would still be gone.
B. Wrong: Kinesis does not keep records until they are read, so moving the checkpoint has no effect on expired records. The consumer could only restart from whatever the retention window still holds.
C. Wrong: Enhanced fan-out changes read throughput for each consumer, not how long records stay in the stream. The reporting job still could not read records that have already expired.
D. Right: Correct. Kinesis Data Streams keeps records for 24 hours by default, and extended retention lets the stream hold them for up to 365 days. With at least three days of retention, the reporting consumer can restart from a timestamp inside that window after a future fix. Learn more: Change the data retention period (opens in a new tab)
Explanation
A replay can only read records that the stream still holds. The default retention is 24 hours, so being able to replay three days needs extended retention set to cover that window; records that already expired cannot be recovered. Shard count, checkpoints and fan-out do not change how long records are kept.
Want more on Throttling, fan-out and replay? The 5 full-length practice sets have 10 more Throttling, fan-out and replay questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →