Incident Response for Ops
Runbooks, escalation, postmortems, and blameless culture. What to do when things go sideways.
“The first rule of incident response: don't make it worse. The second rule: write down what happened so you don't do it again.”
What you'll learn
First job: stop making it worse
An incident is an unplanned drop in the thing you promised. The first moves are: declare it, name a commander, and keep a timeline. The commander talks to the room. Everyone else works or writes. Nobody 'quickly restarts prod' without saying so in the channel.
If you do not know whether it is an incident, it is an incident until a commander says it is not. Ambiguity is how four people apply four fixes.
Runbooks: write them on a calm day
A runbook is symptoms, likely causes, diagnostic commands, and the reversible next step. It is not a novel and it is not a vibe. Link it from the alert. If the alert pages and the runbook is a Confluence stub from 2022, you have decorated a pager.
Common first pages: 'error rate high', 'deploy looks stuck', 'certificate expires in 48 hours', 'disk will fill'. Write those four before you write a runbook for a failure you have never seen.
Escalation is a map, not a feeling
SEV1: customers cannot do the core action. Wake people. SEV2: a major path is degraded. SEV3: a workaround exists. SEV4: it can wait for business hours. Write the definitions. Then write who is next if the primary does not ack in N minutes.
Escalating is not failure. Silence is. An on-call rotation without a backup is a single point of failure with a calendar.
During: command, comms, and the urge to fix forward
Incident command is a role, not a personality. Comms (status page, account team, leadership) is a different role if you have the people. Engineering investigates. Do not combine all three in one exhausted person unless the company is three people.
Prefer rollback or traffic shift over a clever patch you thought of in Slack. Fix-forward is allowed when rollback is worse. Say that out loud so the room can disagree.
After: the blameless writeup is the product
A postmortem is what happened, when, what made it worse, what made it better, and the changes that make this shape of failure harder. No villains. If the story needs a villain, you are not done editing.
Action items need an owner and a date. 'We should improve monitoring' is not an action item. 'Add a runbook link to alert X and a disk-fill forecast' is. Share the writeup with the people who got paged and the people who write the code. Then do the work, or do not bother writing it down.