Safe remediation, auto scaling and automatic recovery: SAP-C03 study notes and practice questions
What to know about Safe remediation, auto scaling and automatic recovery for the SAP-C03 exam, from the Operational Excellence and Automation domain (20% of the exam guide), then 2 free questions to test it. Part of the SAP-C03 study guide.
SAP-C03 asks how to scale ahead of demand, how to recover from failures without people, and how to make sure automated actions, including retries, don’t make an outage worse.
Safety patterns
Pattern
What it does
AWS examples
Retries with exponential backoff and jitter
Spreads retries out so clients don’t retry in lockstep
AWS SDK retry modes; Step Functions Retry with BackoffRate, MaxDelaySeconds and jitter
Circuit breaker
Stops calling a failing dependency or stops a bad rollout
ECS deployment circuit breaker; application-level breakers with fallbacks
Rate limit
Caps how fast work or changes happen
API Gateway throttling and usage plans; Lambda reserved concurrency to protect a database; Automation concurrency and error thresholds
Bulkhead
Isolates resources so one failure can’t take everything
Separate queues, functions or cells per tenant or feature; reserved concurrency per function
Idempotency
Makes a retried action safe
Idempotency keys, conditional writes
Retries without backoff, or retries at every layer, multiply load on a struggling service.
Auto scaling policies
Target tracking: keep a metric at a value (CPU 50%, requests per target); the default choice.
Step scaling: different adjustments for different alarm breach sizes.
Scheduled scaling: known events, such as business hours.
Predictive scaling: forecasts from daily and weekly patterns and launches capacity before the load arrives; start in forecast only mode to check the forecast, and combine with target tracking for surprises.
Warm pools keep pre-initialized instances for slow-starting apps.
Application Auto Scaling covers ECS services, DynamoDB, Aurora replicas, Lambda provisioned concurrency and SageMaker AI endpoints.
Automatic recovery
Auto Scaling groups replace unhealthy instances; turn on ELB health checks so an instance that fails the application check (not just the EC2 status check) is replaced, with a grace period for startup.
EC2 automatic recovery moves an instance to new hardware after a system status check failure.
ECS services replace stopped or unhealthy tasks to keep the desired count; container health checks and load balancer health checks both count.
Lambda asynchronous invocations retry (up to two retries by default, configurable) and then send the event to an on-failure destination or dead-letter queue; set a maximum event age.
SQS event sources return messages after the visibility timeout; use partial batch responses and a DLQ with maxReceiveCount. Kinesis and DynamoDB streams support bisecting a failed batch, a retry limit and an on-failure destination.
How to choose
Scenario wording
Choose
“Traffic rises every weekday at 8 a.m.; instances take 10 minutes to boot”
Predictive scaling (plus warm pools)
“Keep average CPU at 60%”
Target tracking
“Instances pass EC2 checks but the app returns 500s and aren’t replaced”
ELB health checks on the Auto Scaling group
“A Lambda function overwhelms the database during spikes”
Reserved concurrency or an SQS buffer
“Failed async events must not be lost”
Lambda on-failure destination
“One poison message blocks the batch”
Partial batch responses and a DLQ
Common traps
Immediate retries with no backoff, causing a retry storm.
EC2-only health checks on an Auto Scaling group behind a load balancer.
Predictive scaling for random, unpatterned spikes.
Unbounded Lambda concurrency in front of a fixed-size database.
Automated remediation with no limit, which can terminate a whole fleet on a false alarm.
A payment processor uses an automated remediation to restart application instances when a health alarm fires. During a database slowdown, the remediation restarted hundreds of instances within minutes, which made the outage worse and overloaded the database when the instances reconnected.
The operations team wants the remediation to keep repairing single-instance failures, but to stop acting when failures are widespread, and it wants every automated action limited in rate and recorded for review. A person must be notified when the automation stops.
Which design meets these requirements?
Choose one answer.
AA. Keep the restart rule as it is, and add a second EventBridge rule that restarts the instances again whenever the first restart fails, so that every instance is eventually repaired without a person.
BB. Start a Systems Manager Automation runbook from EventBridge with maximum concurrency and maximum errors set, and have it check a DynamoDB counter that opens a circuit breaker and notifies on-call.
CC. Remove the automatic restart, and have an engineer restart each failed instance by hand from the console as soon as the alarm fires.
DD. Replace the restart with an automatic scale-out of the fleet by 50 percent whenever any health alarm fires, so that the healthy instances take the extra work.
Show the answer and explanation
Answer: B
A. Wrong: A second restart rule adds more restarts during the same incident and makes the overload worse. The option has no limit on rate and no stop when failures are widespread.
B. Right: Correct. The runbook limits how many instances run at once and stops on errors, and the counter in DynamoDB holds a circuit breaker that halts restarts when failures are widespread. The on-call team is notified, and each action is recorded in the automation history.
C. Wrong: A manual restart is safe but slow, and it does not repair single-instance failures automatically. The team wants the automation to keep working for those cases.
D. Wrong: Scaling out during a database slowdown adds more connections to the overloaded database. The option does not limit the actions or stop them when failures are widespread.
Explanation
A Systems Manager runbook with concurrency and error limits, plus a DynamoDB circuit breaker that halts restarts and notifies on-call when failures are widespread, keeps single-instance repairs while preventing a restart storm. Extra restart rules, manual-only repair and scale-out during a database slowdown each miss a safety requirement.
A news site's traffic rises sharply every weekday at 7 AM when the morning newsletter goes out, and the spike lasts about an hour. The web tier runs on an Auto Scaling group, and the team currently scales only on CPU, so the group reacts after the pages are already slow.
The team wants capacity ready before the spike each morning, and it still wants the group to follow unexpected traffic during the rest of the day without over-provisioning.
Which scaling configuration meets these requirements at the LOWEST cost?
Choose one answer.
AA. Set the group's minimum and desired capacity to the peak number of instances all day, so that the fleet is always large enough for the morning spike.
BB. Add a scheduled action that raises the minimum capacity before 7 AM each weekday, and keep a target tracking policy so the group follows unexpected traffic during the rest of the day.
CC. Add a step scaling policy that adds instances when average CPU passes 60 percent and removes them when CPU falls below 30 percent, with larger steps for sharper rises in load and no schedule.
DD. Run the web tier on a larger instance type, so each instance handles the morning spike without any scaling.
Show the answer and explanation
Answer: B
A. Wrong: Keeping the peak capacity all day pays for idle instances for most of the day. The option removes the scaling that the team needs for the rest of the day.
B. Right: Correct. The scheduled action has capacity in place before the newsletter, and target tracking adds or removes instances for other traffic. The fleet is sized to demand at each time of day, so the cost stays low.
C. Wrong: A step policy reacts after CPU rises, so the pages are slow again before the new instances are ready. The option does not prepare the capacity before the spike.
D. Wrong: Larger instances cost more every hour, including the quiet hours. The option does not follow demand and raises the cost.
Explanation
A scheduled action prepares capacity before the known weekday spike, and target tracking follows unplanned traffic, so the group is sized to demand at each time of day. Constant peak capacity, reactive step scaling and larger instances each pay for idle capacity or respond too late.
Want more on Safe remediation, auto scaling and automatic recovery? The 5 full-length practice sets have 8 more Safe remediation, auto scaling and automatic recovery questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →
Study Safe remediation, auto scaling and automatic recovery