Streaming data with Kinesis: DVA-C03 study notes and practice questions
What to know about Streaming data with Kinesis for the DVA-C03 exam, from the Development with AWS Services domain (30% of the exam guide), then 2 free questions to test it. Part of the DVA-C03 study guide.
Streaming is covered in Domain 1 under managing streaming data with AWS services. Questions describe a continuous feed, such as meter readings, clickstreams or orders, and ask how to produce, store and consume it reliably.
Choosing the service
Kinesis Data Streams keeps an ordered sequence of records in shards. Consumers read them, can replay them within the retention period, and can process them in real time.
Amazon Data Firehose buffers records and delivers them in batches to destinations such as S3, OpenSearch or Redshift. It does not replay records from the stream. Choose it for loading data, not for application consumers.
Lambda reads Kinesis through an event source mapping that polls the stream for the function.
Producing records
Each record has a partition key. Kinesis hashes it to choose a shard, so records with the same key stay in order on one shard.
A key with few distinct values, such as a country code, creates hot shards however many shards the stream has. Choose a key with many distinct values, such as a meter ID.
PutRecords can return HTTP 200 while some records fail. Read FailedRecordCount and resend only the failed entries, matched by position.
Consuming records
Shared throughput: consumers that use GetRecords share each shard’s read throughput. More consumers mean more latency and throttling.
Enhanced fan-out: each registered consumer gets its own read throughput per shard, and records are pushed to it. Use it when several applications read the same stream at low latency.
Retention is 24 hours by default and can be raised for replay, with extra cost above the default.
Lambda event source mapping settings
Problem in the question
Setting
One bad record blocks its shard and keeps retrying
BisectBatchOnFunctionError to split the batch, a maximum retry count, and an on-failure destination
Backlog is too slow on one shard, and ordering by key must still hold
ParallelizationFactor to run more batches per shard
Function should start from existing backlog, not only new records
StartingPosition set to TRIM_HORIZON; LATEST skips the backlog
Records are too old to be worth processing
Maximum record age
Low volume and invocations are too frequent
A larger batching window with a bigger batch size
Windows and state
Tumbling windows aggregate records over a time period. Lambda limits the state kept per shard for each window, and ends the window early if the state grows past the limit. Keep state compact.
DynamoDB Streams
A stream record’s view type decides what it contains. NEW_AND_OLD_IMAGES includes both the previous and the new item, which an audit trail needs.
Common traps
Low-cardinality partition keys for write-heavy streams.
Adding consumers to a shared-throughput stream when the question needs low latency. Use enhanced fan-out.
Infinite retries on a Kinesis-triggered function with no bisect or destination.
Firehose for replay or for several independent application consumers.
Extending retention when the problem is failed records. Retention keeps data; it does not fix errors.
A smart-meter platform writes readings to an Amazon Kinesis data stream with 20 shards.
Each producer sets the partition key to the region name, and there are only four regions. The stream shows write throttling on a few shards, while the other shards are nearly idle and total traffic is far below the stream capacity.
What should the developer change?
Choose one answer.
AA. Use the meter ID as the partition key, since every meter has its own value.
BB. Double the stream to 40 shards, so that each shard receives fewer records.
CC. Change the producers from PutRecords to PutRecord, so that each record is sent in its own call.
DD. Use the region name together with the year and month as the partition key, so the key changes every month.
B. Wrong: With only four partition key values, the records still land on at most four shards. The extra shards would sit idle.
C. Wrong: One record per call changes how requests are batched, not which shard each record goes to. The same few shards stay overloaded.
D. Wrong: This adds little variety. In any given month there are still only four key values, so the same few shards remain hot.
Explanation
Kinesis hashes each partition key to choose a shard, so all records with the same key go to the same shard. Four key values mean at most four busy shards, however many shards the stream has. A key with many distinct values spreads the writes evenly across the stream.
A Lambda function reads an Amazon Kinesis data stream through an event source mapping with default settings.
One malformed record makes each attempt fail, so its shard stops advancing and the records behind it wait. The team wants the rest of the data to keep flowing, and it wants the malformed records kept for later investigation.
Which TWO changes should the developer make? (Choose two.)
Choose two answers.
AA. Turn on BisectBatchOnFunctionError, so that a failing batch is split in half and retried.
BB. Set a finite MaximumRetryAttempts and an on-failure destination such as an SQS queue.
CC. Configure a dead-letter queue in the function's asynchronous invocation settings.
DD. Increase the visibility timeout on the stream so the function has longer per batch.
EE. Add more shards so that the failing shard receives fewer records.
C. Wrong: That setting applies to asynchronous invocations. A stream event source mapping uses its own on-failure destination instead.
D. Wrong: Visibility timeout is an Amazon SQS setting. Kinesis has no such setting, and a longer wait would not remove the bad record from the shard.
E. Wrong: Resharding does not remove the malformed record. The record stays in its shard, which still stalls until the record is handled.
Explanation
With default settings a stream mapping retries a failing batch without limit, so one bad record blocks its shard. Bisecting the batch isolates the bad record, and a finite retry count with an on-failure destination lets Lambda discard it, keep the shard moving and keep the details for review.
Want more on Streaming data with Kinesis? The 5 full-length practice sets have 10 more Streaming data with Kinesis questions, 390 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →