Exams / SCS-C03 / Study guide / Analyzing and correlating logs (CloudWatch Logs Insights, Athena, OpenSearch)
Analyzing and correlating logs (CloudWatch Logs Insights, Athena, OpenSearch): SCS-C03 study notes and practice questions
What to know about Analyzing and correlating logs (CloudWatch Logs Insights, Athena, OpenSearch) for the SCS-C03 exam, from the Detection domain (16% of the exam guide), then 1 free question to test it. Part of the SCS-C03 study guide.
These questions describe an analyst who must answer a question about logs: which role made a call, which address reached an instance, or how many failed logins happened last quarter. You pick the query service by where the logs are stored, how far back the question goes, and how much the query costs to run.
Choosing a query service
CloudWatch Logs Insights queries one or more log groups with its own query language, OpenSearch PPL or OpenSearch SQL. It suits recent logs and quick troubleshooting. With cross-account observability, a monitoring account can query log groups in linked source accounts and Regions.
Amazon Athena runs standard SQL on data in S3 with no servers to manage, using table definitions in the AWS Glue Data Catalog. It suits long-term logs such as CloudTrail, VPC Flow Logs and load balancer access logs. You pay for the data each query scans.
Amazon OpenSearch Service indexes logs for full-text search, dashboards and alerting. It suits interactive investigation over large volumes, and it costs more to run than a query on files in S3.
Joins across sources, such as CloudTrail identities matched to flow log addresses, need SQL. Logs Insights can join log groups with OpenSearch SQL while the data is still in CloudWatch Logs. For long history stored in S3, use Athena.
Cost and speed
Partition Athena tables by account, Region and date, and use partition projection, so a query for one day scans one day of files.
Store logs as Parquet or another columnar format when you query them often, because Athena then reads only the columns it needs.
Keep CloudWatch Logs retention in line with the investigation window. Logs deleted after retention expires cannot be queried, so export older logs to S3 first.
How to choose
Scenario wording
Pick
Errors in a CloudWatch log group from the last hour
Logs Insights
Join CloudTrail and VPC Flow Logs stored in S3 for the last 90 days
Athena with partitioned tables
Query logs from many accounts from one central account
Logs Insights with cross-account observability, or Athena on the central bucket
Interactive search and dashboards over large log volumes
OpenSearch Service
Keep cost low for old logs that are rarely queried
Athena over partitioned, compressed or Parquet files in S3
Map which resources an IAM role touched and how
Amazon Detective (see the investigation topic)
Common traps
Querying all of S3 without partitions. The query is slow and expensive because it scans everything.
Using Logs Insights for years of history after the logs have expired from CloudWatch Logs.
Standing up an OpenSearch domain for a one-time question that one Athena query answers.
Querying an S3 archive with Logs Insights. It reads log groups only; logs exported to S3 need Athena.
Forgetting the result location. Athena writes results to an S3 location that the user must be able to write to.
A payments processor sends its AWS CloudTrail events to a CloudWatch Logs log group. During an incident review, the security engineer must find which IAM principals made the most failed API calls in the last 24 hours, with the principal and the count shown for each.
The logs are already in CloudWatch Logs, and the answer is needed within minutes. The engineer must not copy or export the logs or build a new pipeline, and must not miss failures with error codes other than AccessDenied.
Which TWO query steps should the security engineer include in a CloudWatch Logs Insights query? (Choose two.)
Choose two answers.
AA. Group the events by sourceIPAddress instead, to see which network addresses made the most failed calls.
BB. Export the log group to an Amazon S3 bucket, then query the exported files with Amazon Athena to count the failed calls by principal.
CC. Filter the events to those where the errorCode field is present, using a filter ispresent(errorCode) statement.
DD. Filter the events where errorCode equals AccessDenied, since that is the most common failure that attackers trigger.
EE. Group the remaining events by userIdentity.arn with a stats count statement, sorted in descending order, so that the most active principals appear first.
Show the answer and explanation
Answers: C, E
A. Wrong: Source addresses show where calls came from, but the requirement asks which principal made them. One address can be shared by several principals, so this grouping gives the wrong answer.
B. Wrong: Exporting copies the logs, which the requirement rules out, and exports are not real time. The answer would arrive too late for the minutes-long window that the review needs.
D. Wrong: This filter finds only one failure code. Calls that return other codes, such as UnauthorizedOperation or InvalidClientTokenId, would be missed, which breaks the requirement to see all failures.
E. Right: Correct. A stats count grouped by userIdentity.arn returns one row for each principal with its number of failed calls. Sorting in descending order puts the principals with the most failures at the top.
Explanation
A Logs Insights query first keeps only failed calls, which CloudTrail marks with an errorCode field, and then counts them by principal. Together these steps answer the question in place, without copies or pipelines. Grouping by source address, exporting the logs and filtering on one error code each fail a requirement.
Want more on Analyzing and correlating logs (CloudWatch Logs Insights, Athena, OpenSearch)? The 5 full-length practice sets have 7 more Analyzing and correlating logs (CloudWatch Logs Insights, Athena, OpenSearch) questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →
Study Analyzing and correlating logs (CloudWatch Logs Insights, Athena, OpenSearch)