Root Cause Analysis (RCA)

A root cause analysis (RCA) is the structured investigation you do after an incident is mitigated, to find out why the system failed and how to stop it happening again. A good RCA goes past "there was a bug" to the gap in tests, process or design that let the bug reach customers. It usually ends in a blameless postmortem document with owned action items.

The process at a glance#

An RCA follows the same fourteen steps every time.

Step	Goal
1. Stabilize first	Investigate only after impact has stopped
2. Preserve evidence	Save logs, metrics and dumps before they disappear
3. Build the timeline	Know exactly what happened when
4. Define the problem	State precisely what failed and what didn't
5. Correlate with changes	Find candidate triggers
6–7. Form and test hypotheses	Eliminate explanations until one fits everything
8. 5 Whys	Dig from the code bug to the system gap
9. Separate the layers	Trigger vs root cause vs contributing factors
10. Validate	Prove the cause explains all symptoms
11. Action items	Fix, prevent, detect, mitigate faster, limit blast radius
12–14. Write, review, track	Share the learnings and close the loop

Step 1: Stabilize first#

Don't start digging while customers are still affected.

RCA starts after mitigation, once service is restored and metrics are back to baseline (see the on-call process).
Exception: if mitigation keeps failing, you may need a partial RCA to find the right mitigation. Keep it narrowly focused.

Step 2: Preserve evidence#

Grab the evidence before it ages out or gets cleaned up.

Save logs, metrics, traces and dashboards for the incident window before retention expires.
Capture heap dumps, thread dumps or core dumps if relevant.
Snapshot configs, feature-flag states and deployed versions at incident time.
Don't restart or clean up hosts you may still need to inspect.
Why: evidence disappears fast, and you can't reconstruct it later.

Step 3: Build the timeline#

Collect timestamps for:

Trigger: the deploy, config change or event that started it
Start of impact: the first bad metric
Detection: when the alarm fired or someone noticed
Acknowledgment: when on-call picked it up
Mitigation: when impact stopped
Resolution: when the permanent fix shipped

Use deploy logs, alarm history, incident-channel messages and metric graphs. Put everything in one time zone (UTC). Gaps in the timeline show where detection or response was slow.

Step 4: Define the problem precisely#

A precise problem statement keeps the investigation focused.

What failed, in specific terms (e.g. "checkout API returned 500s for 18% of requests").
Scope: which regions, hosts, customers, endpoints.
What was not affected. This narrows the search a lot.
When it started and stopped.

Step 5: Correlate with changes#

Line up the start of impact against:

code deploys
config and feature-flag changes
infrastructure changes (scaling, instance types, OS patches)
dependency changes (upstream deploys, third-party outages)
traffic changes (spikes, a new client, bot traffic)
time-based triggers (cron jobs, certificate expiry, TTL expiry, daylight saving)

A strong correlation is a lead, not proof.

Step 6: Form hypotheses#

Turn the evidence into a short list of candidate causes.

List possible causes based on the evidence.
For each, ask: what would you expect to see if this were true?
Rank by likelihood and by how cheap it is to test.

Step 7: Test hypotheses#

Check each candidate against the evidence, cheapest test first.

Look in logs and traces for the predicted signature.
Compare a failing request with a successful one.
Compare affected hosts or regions with unaffected ones.
Reproduce in staging or locally if you can.
Diff the bad version against the last good one.
Rule causes out as much as in; eliminate until one explanation fits all the evidence.

The real root cause must explain every symptom, including why some things weren't affected.

Step 8: Dig to the root with the 5 Whys#

Keep asking "why" until you reach a process or system gap, not just a code bug.

Why did checkout fail? → The payment service threw null pointer exceptions.
Why? → A new upstream field was sometimes null.
Why wasn't it handled? → The code assumed the field always existed.
Why wasn't that caught? → No test covered a missing field.
Why? → There were no contract tests with the upstream service.

Stop when the answer is something you can fix systemically.

Step 9: Separate the layers of cause#

An incident usually has several layers of cause, and each needs its own fix.

Layer	Example
Trigger	The deploy
Root cause	Missing null handling (and no contract tests)
Contributing factors	Missing or late alarm; no canary, so it hit 100% of traffic at once; unclear runbook; a retry storm amplified load
Detection gap	Why it took X minutes to notice

There is often more than one root cause. List them all.

Step 10: Validate the root cause#

Before you write it down, prove the cause is the real one.

Does fixing it provably prevent the failure? Test the fix against your reproduction.
Does it explain the timing, the scope and the parts that weren't affected?
If any symptom is unexplained, you're not done.

Step 11: Define action items#

Write at least one per cause and contributing factor, grouped by type:

Type	Examples
Fix	The immediate code or config correction
Prevent	Tests, input validation, contract tests, schema checks
Detect	New or better alarms, tighter thresholds, synthetic checks
Mitigate faster	Runbook updates, auto-rollback, feature flags
Limit blast radius	Canary deployments, cell-based architecture, circuit breakers, rate limits

Each item gets an owner, a priority and a due date. Prefer systemic fixes; "be more careful" is not an action item.

Step 12: Write the RCA / postmortem document#

The document captures what happened and what will change. It usually covers:

Summary
Customer impact (who, how many, how long)
Timeline
Root cause and contributing factors
5 Whys
Metrics: time to detect (MTTD), time to mitigate, time to resolve (MTTR)
What went well, what went poorly, where you got lucky
Action items table

Keep it blameless: describe what the system allowed, not who made a mistake.

An RCA is only useful if others read and question it.

Review with the team and stakeholders.
Challenge the conclusion: is this the real root cause, or just the nearest one?
Share learnings with other teams that might have the same weakness.

Step 14: Track to closure#

The RCA isn't finished until its action items are.

Follow action items until they're done.
Verify the fixes work (e.g. the new alarm actually fires in a test).
If a similar incident happens again, revisit the RCA; it missed something.

Key takeaways#

Mitigate first, investigate second, and preserve evidence early.
A root cause must explain all symptoms, including what wasn't affected.
The trigger is not the root cause; go past the code bug to the system gap.
Blameless: systems fail, not people.
No action items means no RCA, and "be more careful" is never a fix.