Real-time and batch analytics (Kinesis, MSK, EMR, Redshift): SAP-C03 study notes and practice questions
What to know about Real-time and batch analytics (Kinesis, MSK, EMR, Redshift) for the SAP-C03 exam, from the Architecture Design and Implementation Built for the Cloud domain (25% of the exam guide), then 1 free question to test it. Part of the SAP-C03 study guide.
SAP-C03 describes data arriving at some rate from some source and asks for the pipeline that delivers results in the required time with the least management.
Streaming
Service
Fits
Kinesis Data Streams
AWS-native streams, ordered by partition key, replayable; on-demand or provisioned capacity
Amazon MSK
Apache Kafka compatibility for existing Kafka apps and connectors; provisioned (with Express brokers) or MSK Serverless; MSK Connect for connectors
Amazon Data Firehose
Load streams into S3, Redshift, OpenSearch, Splunk or HTTP endpoints with buffering, format conversion to Parquet and Lambda transforms; near real time, no code
Managed Service for Apache Flink
Stateful stream processing: windows, joins, anomaly detection, exactly-once state
Batch and big data
Amazon EMR: Spark, Hive, Presto, Trino and HBase on EC2 (most control, Spot for task nodes), EKS or EMR Serverless (no clusters to size). Use transient clusters for scheduled jobs and keep data in S3, not HDFS.
AWS Glue: serverless Spark ETL with the Data Catalog; good for regular ETL without cluster tuning.
Warehousing
Amazon Redshift: columnar MPP warehouse; RA3 nodes separate compute and managed storage; Redshift Serverless for variable or unpredictable usage; concurrency scaling for bursts; materialized views for repeat queries.
Zero-ETL integrations replicate data from Aurora, RDS, DynamoDB and some SaaS apps into Redshift without building pipelines.
Streaming ingestion reads Kinesis or MSK directly into materialized views.
SaaS data
Amazon AppFlow pulls data from SaaS apps (such as Salesforce, ServiceNow, Zendesk) into S3 or Redshift on a schedule, on an event or on demand, with field mapping, filtering and PrivateLink where supported.
How to choose
Scenario wording
Choose
“Existing Kafka producers and consumers, minimal changes”
Amazon MSK
“Load clickstream into S3 as Parquet with no code”
Amazon Data Firehose
“Detect fraud patterns over 5-minute windows in real time”
Managed Service for Apache Flink
“Nightly Spark jobs, no cluster management”
EMR Serverless (or Glue)
“BI dashboards on petabytes of structured data”
Redshift
“Analyze Aurora data in the warehouse without ETL pipelines”
Zero-ETL integration with Redshift
“Sync Salesforce records to S3 daily”
AppFlow
Common traps
Lambda for long stateful stream processing that Flink handles natively.
Long-running EMR clusters storing data in HDFS for jobs that run once a day.
Self-managed Kafka on EC2 when MSK fits.
Kinesis Data Streams when nothing reads the stream except a loader to S3; Firehose alone is simpler.
Custom ETL from Aurora to Redshift when zero-ETL meets the need.
A news site's clickstream produces about 50,000 events a second at peak. Editors need article view counts on a dashboard that is never more than about one minute behind, and the data team needs the raw events kept in Amazon S3 for seven years.
The dashboards run on Amazon Redshift, and the platform team wants the pipeline to keep working when traffic spikes, without a fleet of servers to manage.
Which solution meets these requirements?
Choose one answer.
AA. Send the events to a Kinesis Data Streams stream in on-demand mode, use Redshift streaming ingestion with materialized views for the dashboards, and have a Data Firehose stream archive the raw events to S3.
BB. Have a Lambda function triggered by the event source write each event to Amazon Redshift with a single-row INSERT statement, and have the same function also write each raw event to S3 for the seven-year archive.
CC. Deliver the events to S3 in hourly batches, and run a Redshift COPY job at the end of each hour to load the new files into the dashboard tables.
DD. Store the events in an Amazon EMR cluster running Spark Streaming on a fixed number of core nodes, and have the dashboards query the cluster's output tables.
Show the answer and explanation
Answer: A
A. Right: Correct. Redshift streaming ingestion reads from the stream directly and keeps the dashboard data within about a minute, and an on-demand stream scales with the event rate. Amazon Data Firehose reads the same stream and writes the raw events to S3 for the archive.
B. Wrong: Single-row inserts at 50,000 events a second would overwhelm the cluster, and a spike would cause timeouts and lost writes. The approach is also hard to keep within the one-minute target.
C. Wrong: The hourly batches make the dashboard up to an hour behind, which misses the one-minute target. The design does meet the archive requirement.
D. Wrong: The fixed cluster does not scale with spikes, and it adds servers to manage. The dashboards are also meant to run on Redshift, so this design moves them off the planned platform.
Explanation
An on-demand stream feeding Redshift streaming ingestion meets the one-minute freshness and spike requirements with no servers to run, and Firehose keeps the archive. Per-row inserts, hourly loads and a fixed Spark cluster each fail the latency, scale or operations requirement.
Want more on Real-time and batch analytics (Kinesis, MSK, EMR, Redshift)? The 5 full-length practice sets have 5 more Real-time and batch analytics (Kinesis, MSK, EMR, Redshift) questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →
Study Real-time and batch analytics (Kinesis, MSK, EMR, Redshift)