Streaming and analytics: SAA-C03 study notes and practice questions
What to know about Streaming and analytics for the SAA-C03 exam, from the Design High-Performing Architectures domain (24% of the exam guide), then 2 free questions to test it. Part of the SAA-C03 study guide.
Analytics questions on SAA-C03 aren’t deep, but they expect you to place each service in the pipeline: ingest, store, transform, query.
Ingest
Service
Use for
Key details
Kinesis Data Streams
Real-time streams that several consumers read and can replay
Retention 24 hours by default, up to 365 days. Provisioned shards (1 MB/s or 1,000 records/s in, 2 MB/s out each) or on-demand mode. Enhanced fan-out gives each consumer its own 2 MB/s per shard
Amazon Data Firehose
Delivering a stream to S3, Redshift, OpenSearch, Splunk or HTTP endpoints with no code
Fully managed, buffers by size or time, can convert to Parquet and transform with Lambda. Near real time, not for replay
Amazon MSK
Apache Kafka workloads
Managed Kafka, including a serverless option
AWS DMS
Migrating databases and replicating ongoing changes
Change data capture into other databases, S3 or Kinesis
Store, transform and query
Amazon S3 is the data lake: cheap, durable and readable by every analytics service.
AWS Glue crawls data to build the Glue Data Catalog (table definitions) and runs serverless ETL jobs. AWS Lake Formation adds fine-grained permissions on the lake.
Amazon Athena runs SQL directly on S3, serverless, billed by data scanned. Partitioning (for example by date) and columnar formats such as Parquet cut the data scanned, and so the cost.
Amazon Redshift is the data warehouse for complex, frequent queries over large structured data; Redshift Serverless removes cluster sizing, and Redshift Spectrum queries S3 data from Redshift.
Amazon EMR runs big data frameworks (Spark, Hadoop) when you need that control.
Amazon OpenSearch Service is for log analytics and full-text search.
Amazon QuickSight builds dashboards on top of all of these.
How to choose
“Occasional ad hoc SQL on data already in S3, lowest cost”: Athena.
“Load streaming data into S3 in Parquet with no code”: Firehose.
“Several applications process the same stream within a second and can replay days of data”: Kinesis Data Streams with longer retention and enhanced fan-out.
“Complex BI queries over terabytes, many times a day”: Redshift.
Common traps
Firehose for replay or multiple custom consumers. It delivers; it doesn’t store for re-reading.
Redshift for occasional queries. Paying for a warehouse that sits idle loses to Athena.
Querying raw CSV in Athena at scale when the question mentions cost. Convert to Parquet and partition.
A media company streams about 20 MB/s of viewing events from its apps into AWS.
Three teams each build their own consumer application: live dashboards, fraud detection and recommendations.
Each consumer must read every event within about a second of arrival without slowing the other consumers down. After a bug fix, any team must be able to reprocess the last 3 days of events.
Which solution meets these requirements?
Choose one answer.
AA. Write the events to an Amazon Kinesis data stream with default retention, and register each team's application as an enhanced fan-out consumer.
BB. Write the events to an Amazon Kinesis data stream with a 72-hour retention period, and have all three applications read each shard with GetRecords.
CC. Write the events to an Amazon Kinesis data stream with a 72-hour retention period, and register each team's application as an enhanced fan-out consumer.
DD. Publish the events to an Amazon SNS topic that fans out to one Amazon SQS queue per team, with each queue's message retention set to 4 days.
Show the answer and explanation
Answer: C
A. Wrong: Enhanced fan-out gives each registered consumer its own 2 MB/s per shard, so the consumers do not compete. However, the default retention period is 24 hours, so a team could not go back 3 days after a bug fix. The retention period must be increased to at least 72 hours. Learn more: Change the data retention period (opens in a new tab)
B. Wrong: The 72-hour retention meets the replay need, but standard GetRecords consumers share a shard's 2 MB/s read limit. Each shard takes up to 1 MB/s of writes, so three consumers would need up to 3 MB/s per shard and would slow each other down. Enhanced fan-out gives each consumer dedicated throughput. Learn more: Quotas and limits (opens in a new tab)
C. Right: Correct. A 72-hour retention period keeps 3 days of records available for any consumer to re-read from an earlier position. Enhanced fan-out pushes records to each registered consumer with its own 2 MB/s per shard, typically in about 70 ms, so the three teams do not affect each other. Learn more: Develop enhanced fan-out consumers with dedicated throughput (opens in a new tab)
D. Wrong: SNS-to-SQS fan-out gives each team its own copy of every event. But a consumer deletes each SQS message once it has processed it, so after a bug fix there is nothing left in the queue to replay. A stream with retention lets consumers re-read records they have already processed. Learn more: Common Amazon SNS scenarios (opens in a new tab)
Explanation
Two limits decide this. Kinesis keeps records for 24 hours by default, and retention can be raised up to 365 days, so 3-day replay needs at least 72 hours. Without enhanced fan-out, all consumers share 2 MB/s of reads per shard while each shard can take 1 MB/s of writes, so three full-speed consumers would compete. Enhanced fan-out gives each registered consumer its own 2 MB/s per shard. SQS queues cannot replay messages that have been processed and deleted.
A research team stores 40 TB of application logs in Amazon S3 as Parquet files, partitioned by date.
Analysts run a few ad hoc SQL queries each week, and most queries look at only one or two days of data. The company does not want to copy the data into another data store, and it wants to pay nothing for compute when no queries are running.
Which solution is the MOST cost-effective?
Choose one answer.
AA. Define the data as a partitioned table in the AWS Glue Data Catalog, and query it in place with Amazon Athena.
BB. Load the files into a provisioned Amazon Redshift cluster with the COPY command, and run the analysts' SQL queries against the cluster's tables.
CC. Create external tables in a provisioned Amazon Redshift cluster, and query the Parquet files where they are with Amazon Redshift Spectrum.
DD. Launch a long-running Amazon EMR cluster with Trino installed, and query the Parquet files in S3 from the cluster when needed.
Show the answer and explanation
Answer: A
A. Right: Correct. Athena is serverless and queries data where it sits in S3, billing by the data each query scans. Filtering on the date partition means only the matching days are read, and Parquet lets Athena read only the needed columns, so a few small queries a week cost very little. Learn more: Optimize data (opens in a new tab)
B. Wrong: A Redshift cluster gives fast SQL over large data sets, but COPY loads the data into Redshift, which breaks the no-copy requirement. A provisioned cluster is also billed for every hour it runs, even in the many hours between the weekly queries. Learn more: Amazon Redshift Spectrum (opens in a new tab)
C. Wrong: Redshift Spectrum queries files in S3 without loading them, so it meets the no-copy requirement. However, it runs through a Redshift cluster, and a provisioned cluster is billed while it runs whether or not anyone is querying, which is costly for a few queries a week. Learn more: Amazon Redshift Spectrum (opens in a new tab)
D. Wrong: Trino on EMR can query Parquet files in S3 in place, so no copy is needed. But a long-running cluster is billed for its instances all week and needs to be sized and maintained, which costs far more than a serverless service for occasional queries. Learn more: Trino on Amazon EMR (opens in a new tab)
Explanation
For occasional SQL over data already in S3, Amazon Athena is the cheapest fit: there is nothing to run between queries, and each query is charged by the data it scans. Partitioning by date and storing Parquet both cut the data scanned, so queries over one or two days are small. The Redshift and EMR options either copy the data or keep a cluster running all week.
Want more on Streaming and analytics? The 5 full-length practice sets have 13 more Streaming and analytics questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →