Fault-tolerant and self-healing architectures: SAP-C03 study notes and practice questions

What to know for the exam

SAP-C03 gives you an architecture with a weak spot (one instance, one AZ, synchronous calls that cascade, a shared component every customer depends on) and asks for the change that makes it tolerate failure with the least effort.

Remove single points of failure

Static stability: run enough capacity in each AZ that losing one AZ doesn’t require launching new resources to carry the load (for example, three AZs at 50% each). Recovery that depends on control plane calls during an event is fragile.

Self-healing

• ELB health checks on the Auto Scaling group replace instances that fail application checks, not just EC2 status checks.
• ECS services and Kubernetes Deployments replace failed tasks and pods; liveness and readiness probes keep traffic off bad ones.
• EC2 auto recovery restarts an instance on new hardware for a single, irreplaceable instance.
• Aurora and RDS Multi-AZ fail over automatically; applications reconnect through the cluster or instance endpoint (use RDS Proxy to shorten reconnects).
• Lambda async retries with dead-letter queues or on-failure destinations; SQS redrive to a DLQ after max receives.
• ARC zonal autoshift moves traffic away from an impaired AZ automatically.

Limiting blast radius

• Decouple with SQS, SNS, EventBridge or Kinesis so a slow consumer doesn’t break producers; queues absorb spikes (load leveling).
• Timeouts on every remote call, and retries with exponential backoff and jitter, capped, only for idempotent operations.
• Circuit breakers stop calling a failing dependency and serve a fallback; ECS deployment circuit breaker rolls back failed deployments.
• Bulkheads: separate pools of resources (connection pools, queues, functions with reserved concurrency) so one noisy workload can’t starve others.
• Cell-based architecture: split customers into independent cells (full copies of the stack) with a thin routing layer, so a failure or bad deployment affects one cell; shuffle sharding reduces overlap between customers’ cells.
• Graceful degradation: serve cached or static content when a dependency is down.
• Throttling at API Gateway and load shedding keep the system up under overload.

How to choose

Common traps

• Retries without backoff, which turn a brownout into an outage.
• EC2 status checks only, so a hung application is never replaced.
• Relying on Auto Scaling to launch capacity during an AZ event.
• Read replicas described as automatic failover for writes on RDS (only Aurora replicas or Multi-AZ fail over automatically).
• Synchronous chains of microservices with no queue or fallback.

Try the 1 practice question ↓