Streaming ingestion with Kinesis and MSK: DEA-C01 study notes and practice questions
What to know about Streaming ingestion with Kinesis and MSK for the DEA-C01 exam, from the Data Ingestion and Transformation domain (34% of the exam guide), then 3 free questions to test it. Part of the DEA-C01 study guide.
Streaming questions describe event volume, latency, consumers and replay needs, then ask which service or setting fits.
The services
Service
Use for
Key details
Kinesis Data Streams
Real-time streams read by several custom consumers, with replay
Shards: 1 MB/s or 1,000 records/s in, 2 MB/s out each (provisioned), or on-demand capacity mode. Retention 24 hours by default, up to 365 days. Ordered per partition key
Amazon Data Firehose
Delivering streams to S3, Redshift, OpenSearch, Splunk, Snowflake, Iceberg tables or HTTP endpoints
Fully managed; buffers by size or time (near real time); can convert to Parquet/ORC, compress, partition by keys and transform with Lambda; no replay
Amazon MSK
Apache Kafka workloads
Keeps the Kafka API, so existing producers and consumers work; provisioned or MSK Serverless; MSK Connect for connectors
Amazon Managed Service for Apache Flink
Stateful stream processing (windows, joins) in SQL, Java or Python
Reads from Kinesis or MSK
Partition keys and hot shards
Records with the same partition key go to the same shard. A key with few values (such as country or event type) sends most traffic to one shard, which throttles (ProvisionedThroughputExceededException) even when total capacity is fine. Use a key with many distinct values, such as a device or user ID.
Common patterns
Replay plus near-real-time delivery: Kinesis Data Streams with extended retention, and Firehose reading from the stream to land data in S3.
Kafka migration: MSK, no client rewrite.
Clickstream to S3 in Parquet with no code: Firehose with format conversion.
Common traps
Firehose for replay or for several custom consumers. It delivers and forgets.
Adding shards to fix a hot shard caused by a bad partition key.
Kinesis for Kafka clients. Use MSK to avoid rewrites.
An esports league runs a match-telemetry platform that publishes about 40,000 events per second from its game servers. Eleven analytics services already consume the stream through Apache Kafka client libraries, and their consumer groups and offsets are tuned.
The league is moving the platform to AWS and does not want to rewrite those consumers. It needs a managed service that keeps the Kafka protocol, topics and partitions, and AWS should handle broker patching and replacement.
Which solution meets these requirements?
Choose one answer.
AA. Create an Amazon MSK cluster, keep the existing Kafka clients, topics and partitions, and let AWS manage the brokers and their patching.
BB. Replace the Kafka clients with the Kinesis Client Library and move every consumer group to a Kinesis Data Streams application with its own offset logic.
CC. Install Kafka on Amazon EC2 instances in an Auto Scaling group, and use AWS Systems Manager Patch Manager to update the brokers on a schedule.
DD. Send the events through Amazon Data Firehose into Amazon S3, and have each consumer read new objects in batches on a schedule.
C. Wrong: This keeps the Kafka protocol, but the league would still run and recover the brokers itself. Patch Manager updates operating systems, not the Kafka cluster, so the operating burden stays with the team.
D. Wrong: Firehose buffers records and delivers them to S3, which adds delay and removes the streaming consumer model. The services would still need to be rewritten to read files instead of streams. Learn more: What is Amazon Data Firehose? (opens in a new tab)
Explanation
Kafka clients should not need rewriting when a platform moves to AWS. Amazon MSK keeps the Kafka API, topics, partitions and consumer offsets, and AWS manages the brokers. Kinesis Data Streams and Firehose are capable services, but each would require changes on the consumer side.
A ski resort group sends lift-scan events from its gates into one Amazon Kinesis Data Streams stream. The producer uses the resort ID as the partition key, so most events come from three large resorts and land on one or two shards.
During holiday weekends, the producers get ProvisionedThroughputExceededException errors on those shards, while the other shards sit almost idle. No scan can be lost, and the group must keep the existing stream and consumers.
Which change meets these requirements?
Choose one answer.
AA. Increase the stream's retention period so producers can retry for longer before the records expire.
BB. Change the partition key to a high-cardinality value, such as gate or device ID, so records spread across shards.
CC. Switch the stream to on-demand capacity mode and keep the resort ID as the partition key.
DD. Turn off producer retries so that throttled writes fail fast, and let each gate keep the rejected records locally.
B. Right: Correct. Kinesis maps each partition key to a shard by hash, so a key with many distinct values spreads the load. The stream and consumers stay the same, and the idle shards start to carry traffic. Learn more: What is Amazon Kinesis Data Streams? (opens in a new tab)
C. Wrong: On-demand mode adds capacity across the stream, but all records with one partition key still go to one shard. That shard's per-shard write limit still applies, so the largest resort keeps getting throttled.
D. Wrong: Failing fast does not spread the load, and the records would sit on gate devices instead of reaching the stream. Scans must not be lost, so this option breaks a stated requirement.
Explanation
A hot shard comes from a partition key with few distinct values, not from the stream's total capacity. Choosing a key with many values, such as a gate or device ID, spreads records across shards and removes the throttling, with no change to the stream or consumers. Retention, on-demand mode and retry changes do not fix the key's distribution.
A logistics broker receives GPS pings from about 6,000 trucks and must land them in an Amazon S3 data lake within about one minute. Analysts query the data by region and by day, and they need Parquet files rather than JSON.
Operations also needs the raw pings kept in a stream for seven days, so that a second consumer can replay them after a bug fix. The team wants no custom consumer code that writes one object per record.
Which TWO options meet these requirements? (Choose two.)
Choose two answers.
AA. Create a Kinesis Data Streams stream with seven-day extended retention, and add a Firehose delivery stream that reads from it into S3.
BB. Write each ping directly to a Firehose delivery stream, and rely on Firehose to keep the pings for seven days so they can be replayed.
CC. Configure a Firehose delivery stream with dynamic partitioning on region and day, and a Glue table schema for Parquet conversion.
DD. Configure a Lambda function that writes one S3 object per ping, keyed by region and date, and expire the objects after one day.
EE. Replace the truck pings with a nightly AWS DMS change data capture task that reads each truck's onboard database and writes Parquet files to S3.
B. Wrong: Firehose delivers records to a destination and does not keep a replayable history, so the seven-day replay requirement is not met. The option works for delivery alone but fails on replay.
C. Right: Correct. Dynamic partitioning writes objects into region and day prefixes, which analysts can filter on. Record format conversion needs a Glue Data Catalog table that describes the schema, and Firehose applies it before writing to S3. Learn more: Enable record format conversion (opens in a new tab)
D. Wrong: One object per ping creates a very large number of small files, which slows analysis and raises request costs. The option also expires the data too soon and does not convert it to Parquet.
E. Wrong: AWS DMS reads from database engines, and trucks do not expose a database to it. A nightly job also misses the one-minute freshness requirement.
Explanation
The seven-day replay and the one-minute delivery need two services working together. A Kinesis stream with extended retention holds the raw pings for replay, and Firehose reads from it to deliver to S3. Dynamic partitioning and a Glue table schema give analysts region and day prefixes in Parquet, without custom code that writes one object per record.
The stream keeps seven days for replay, and Firehose partitions the GPS records by region and day as it lands them in S3.
Want more on Streaming ingestion with Kinesis and MSK? The 5 full-length practice sets have 13 more Streaming ingestion with Kinesis and MSK questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →