Being on call means you are the first responder when a production system you own breaks. Most new engineers join a rotation within their first months. A good on-call shift follows a predictable sequence: prepare before the shift, stop the bleeding when paged, find the root cause once things are stable, and hand off cleanly at the end.
Phase 1: Before the shift#
A few minutes of preparation before the shift saves you from surprises in the middle of the night.
1. Read the handoff#
- Open incidents still being tracked
- Known flaky alarms or ongoing issues
- Workarounds currently in place
- Pending action items from recent postmortems
2. Check the change calendar#
- Deploys in the last 24–48 hours
- Scheduled deploys, migrations or maintenance during your shift
- Changes by upstream and downstream teams
- Why: most incidents are caused by a change, so knowing what changed gives you a head start.
3. Verify your setup#
- Pager app works and notifications are loud
- VPN, production access and permissions work
- Dashboards and log tools load
- You know where the runbooks are
- You know who the secondary on-call and escalation contacts are
- Why: you don't want to discover an access problem at 2 a.m. in the middle of an incident.
Phase 2: When you're paged#
When the pager goes off, work through these steps in order. Restoring service comes before understanding it.
4. Acknowledge#
- Ack within the expected window (often 5–15 minutes).
- Acking signals ownership, so the page doesn't auto-escalate and others don't duplicate effort.
5. Assess impact and set severity#
- Who is affected? All customers, one region, one feature, internal only?
- How bad? Full outage, degraded latency, elevated errors?
- Trend? Stable, growing, or recovering on its own?
Typical severity levels (names vary by company):
| Level | Meaning |
|---|---|
| SEV1 | Major customer-facing outage, all hands |
| SEV2 | Significant degradation, urgent |
| SEV3 | Minor impact, fix during business hours |
Severity decides how fast you move and who you pull in.
6. Rule out a false positive#
- Is the alarm real or a monitoring glitch?
- Cross-check a second signal: another dashboard, customer reports, synthetic checks.
7. Look at what changed#
- Recent deploys: does the error spike line up with a deploy?
- Config or feature-flag changes
- Dependency health: database, cache, downstream services, third-party APIs
- Traffic: sudden spike, bot traffic, unusual load
- Infrastructure: host failures, full disks, memory leaks, expired certificates
8. Mitigate (stop the bleeding)#
The goal is to restore service, not to understand it yet. Options, roughly from safest:
- Roll back the recent deploy (the default)
- Disable the feature flag
- Fail over to a healthy region or replica
- Scale out if it's a capacity problem
- Restart unhealthy hosts or processes
- Throttle or shed load to protect the core path
- Pause the queue and route failures to a dead-letter queue (DLQ) to redrive later
Avoid hotfixing forward under pressure unless rollback isn't possible (for example, after an irreversible database migration).
9. Communicate#
- Open an incident channel or ticket.
- Post an initial message: what's broken, the impact, and what you're doing.
- Update at a fixed cadence (e.g. every 15–30 minutes for a SEV2), even if the update is "still investigating."
- Keep a running timeline of actions and observations; the postmortem needs it.
- For big incidents, split roles: incident commander, communicator, investigators.
10. Escalate when needed#
Escalate if:
- you can't mitigate within a set time window,
- impact is growing,
- it involves a system you don't understand, or
- the fix needs permissions or decisions above you.
Escalate to the secondary on-call, the service owner, the dependency's team or your manager. Escalating early is cheaper than a long outage.
Phase 3: After service is stable#
Once customers are no longer affected, slow down and make sure the incident can't happen the same way again.
11. Verify recovery#
- Error rate, latency and throughput are back to the pre-incident baseline.
- Watch through an observation window (e.g. 30–60 minutes).
- Check for side effects: backed-up queues, failed jobs, inconsistent data.
- Redrive DLQ messages or reprocess failed jobs if needed.
- Announce resolution in the incident channel.
12. Find the root cause#
- Use logs, metrics and distributed traces.
- Build the timeline: when it started, when it was detected, when it was mitigated.
- Apply the 5 Whys to get past the symptom. For example:
- Errors spiked.
- Why? A bad deploy.
- Why did it break? A null field wasn't handled.
- Why wasn't that caught? There was no test for that case.
- Why? There was no contract test with the upstream service.
- Separate the trigger (what set it off) from the root cause (why the system allowed it).
The full method is in Root Cause Analysis.
13. Ship the permanent fix#
- Code or config fix, with tests covering the failure case
- Code review
- Safe rollout: canary first, monitor, then full rollout
14. Write the postmortem#
- Blameless: focus on systems and process, not people.
- Contents:
- summary and customer impact
- timeline
- time to detect (MTTD), time to mitigate, time to resolve (MTTR)
- root cause
- what went well and badly
- action items with owners and due dates
- Review it with the team.
15. Prevent recurrence#
- Detection: add the alarm that would have caught it earlier.
- Noise: delete or tune alarms that fired but didn't help.
- Runbook: add or update steps so the next person resolves it faster.
- Safety nets: better tests, canaries, auto-rollback, circuit breakers, DLQs, rate limits.
- Track action items to completion.
Phase 4: End of shift#
A clean handoff gives the next person everything you know.
16. Hand off#
- Open incidents and their current state
- Temporary mitigations still in place
- Alarms to watch
- Action items in progress
- Anything odd you noticed but didn't resolve
Key takeaways#
- Mitigate before you root-cause: customer impact comes first, and rollback is the default fix.
- Escalate early: there's no prize for solving it alone.
- Over-communicate: silence during an incident creates panic; post updates on a schedule.
- Every alarm must be actionable, or it trains people to ignore alarms.
- Blameless postmortems fix systems, not people, and every incident should make the next one easier.