Treat the database as a stateful production system whose reliability must preserve business correctness as well as uptime.
Treat the database as a stateful production system whose reliability must preserve business correctness as well as uptime.
Module
Foundations
Why this matters in production
Database incidents rarely stay inside the database. They appear to users as failed checkouts, stale reads, duplicated work, timeouts, queue backlog, deployment failure, or lost confidence in data. Database Reliability Mental Model: Correctness, Durability, Availability & Recoverability matters because production reliability requires both system availability and preservation of business correctness.
The goal is not to memorize a vendor command. The goal is to explain the database state, identify the wait/failure boundary, choose a safe action, and prove end-to-end recovery.
Mental model
Text
BUSINESS INVARIANT -> DATABASE STATE -> FAILURE MODE -> EVIDENCE -> RECOVERY
Deep dive
Define reliability across correctness, durability, availability, recoverability, performance and operability.
Separate database uptime from application success: a reachable database can still be unusable because of locks, saturation or stale data.
Identify the business invariants the database must never violate, such as one payment per idempotency key or non-negative inventory.
Map the full path: client pool → proxy → database endpoint → transaction → WAL/redo → storage → replica/backup.
Require evidence for recovery: successful connection is not enough; validate reads, writes, constraints, replication and business transactions.
Use failure budgets and recovery objectives to decide which protections are worth their complexity and cost.
Evidence-first workflow
State the invariant or user symptom. Avoid starting with a CPU graph or a favorite query.
Bound the scope and timeline. Region, tenant, table, endpoint, transaction type, replica and recent changes matter.
Locate the wait or correctness boundary. Connection, lock, query, storage, replication, background maintenance or application behavior.
Collect the smallest useful evidence. Prefer a blocker graph, query plan, pool wait metric, replica position, storage latency or restore proof over broad guessing.
Mitigate amplification. Pause runaway jobs, retries, migrations, backfills or autoscaling behavior before they create more work.
Apply one controlled change. Preserve rollback and diagnostic evidence.
Verify business recovery. A green database dashboard is not enough.
Production example
Text
BUSINESS INVARIANT -> DATABASE STATE -> FAILURE MODE -> EVIDENCE -> RECOVERY
For Database Reliability Mental Model: Correctness, Durability, Availability & Recoverability, write down the expected healthy state and one failure state before changing production.
Failure modes to recognize
Common mistakes:
Treating database reachable as equivalent to application healthy.
Increasing limits before identifying the first saturated resource or wait.
Retrying an operation whose commit outcome may already be successful.
Treating a replica as a backup or a green backup job as a proven restore.
Restarting the database before capturing blocker, query, replication and resource evidence.
Applying several tuning changes simultaneously and losing causality.
Hands-on lab
Production review checklist
What business/user impact does this database behavior create?
Which evidence proves the failure boundary instead of merely correlating with it?
What is the safe rollback or mitigation if the change makes things worse?
How does replication, backup or recovery behavior affect the decision?
Could retries, autoscaling, batch work or connection pools amplify the incident?
What would you monitor to catch this earlier next time?
Interview / production review
Key takeaway
Treat the database as a stateful production system whose reliability must preserve business correctness as well as uptime. A reliable database design is one you can reason about during concurrency, overload, change and failure — and one you have already proven you can recover.