Separate prevention, tolerance and recovery so reliability designs survive real infrastructure, software and data failures.

What you should be able to do

• Explain the reliability/recovery concept in production terms.
• Identify the failure class and evidence that matter for this design.
• Choose a safe HA/DR decision with explicit tradeoffs.
• Validate recovery rather than assuming redundancy is enough.

Deep dive

High availability (HA) and disaster recovery (DR) solve related but different problems. HA tries to keep a service operating while individual components fail. DR restores an acceptable service after a failure exceeds the normal HA design boundary. Backup protects recoverable state; replication keeps another copy current; neither automatically proves that the application can recover.

Model resilience as Failure → Detection → Isolation → Failover/Restore → Validation → Failback/Learning. Every stage has a time budget, an owner and dependencies. A design is incomplete if it describes redundant infrastructure but not how a failed dependency is detected, how traffic moves, how state is reconciled, or how recovery is verified from the user perspective.

The right design starts from business impact. A static documentation site, payroll service and real-time payment platform can have completely different recovery objectives. Reliability engineering is therefore a risk decision, not a contest to build the most expensive architecture.

Commands / calculations / examples

HA protects service continuity inside the normal failure envelope.
DR restores service when the failure exceeds that envelope.
Backup protects recoverable state.
Replication reduces data-distance, but can replicate corruption too.

Production decision rules

• Define the failure envelope before choosing technology.
• Separate service availability from data recoverability.
• A recovery design is not complete until it is tested.