Build a disciplined troubleshooting loop that moves from user impact to evidence, hypothesis, safe test, mitigation and verification.

Troubleshooting model

IMPACT → SCOPE → EVIDENCE → HYPOTHESIS → TEST → MITIGATE → VERIFY

What you should be able to do

• State the customer-facing symptom before naming a component.
• Record exact timestamps and time zones.
• Change one meaningful variable at a time when possible.
• Prefer read-only inspection before restart, delete, failover or scale operations.
• Verify recovery through the critical user path, not only component health.

Deep dive

Production troubleshooting is not a contest to remember the most commands. It is the process of reducing uncertainty without making the incident worse. Start from the user-visible failure, define the scope and time window, collect the smallest useful evidence, form a falsifiable hypothesis, run a safe test, and only then change production state.

A useful troubleshooting statement has four parts: symptom, scope, start time and evidence. “The API is down” is weak. “Checkout POST requests from us-east-1 are returning 503 at 18% since 14:32 UTC while read-only catalog traffic is healthy” is actionable.

Separate observation from interpretation. CPU=95% is an observation. “CPU caused the outage” is an interpretation that still needs proof. Likewise, a pod restart may correlate with recovery without being the root cause.

Prefer reversible, observable actions. During active impact, mitigation can come before complete diagnosis, but capture enough evidence first to avoid destroying the only clues.

Evidence commands / queries

# First-pass evidence examples
uptime
free -h
df -h
ss -s
curl -sv https://service.example.com/health
kubectl get pods -A
kubectl get events -A --sort-by=.lastTimestamp

The command or query output is evidence, not a diagnosis by itself. Always connect it to the question, timestamp, environment and user impact.

Common failure modes

• Jumping to a favorite tool before defining the symptom
• Treating correlation as proof
• Restarting or deleting before capturing volatile evidence
• Testing from a different network, identity or environment than the failing path
• Declaring recovery from component state without verifying the user journey