Runbook: [TITLE]

Category: [category]

Severity: [P1-Critical / P2-High / P3-Medium / P4-Low]

Last Updated: YYYY-MM-DD

Owner: [team/on-call rotation]

Review Cycle: Quarterly

Symptoms

What does this incident look like? What alerts fire? What do users report?

• [ ] Symptom 1
• [ ] Symptom 2
• [ ] Symptom 3

Severity Classification

Detection & Alerts

Diagnosis

Step 1: [First diagnostic action]

# Command to run

Expected output: [what you should see]

If abnormal: [what it means]

Step 2: [Second diagnostic action]

# Command to run

Expected output: [what you should see]

If abnormal: [what it means]

Step 3: [Third diagnostic action]

# Command to run

Mitigation (Stop the Bleeding)

Immediate actions to reduce customer impact before root cause is resolved.

Option A: [First mitigation approach]

# Command to run

Risk: [side effects of this mitigation]

Rollback: [how to undo if it makes things worse]

Option B: [Alternative mitigation]

# Command to run

Resolution

Steps to fully resolve the underlying issue.

Step 1: [Resolution action]

# Command to run

Step 2: [Resolution action]

# Command to run

Verification

How to confirm the issue is truly resolved and not just masked.

• [ ] [Verification check 1]
• [ ] [Verification check 2]
• [ ] [Verification check 3]

# Verification command

Expected: [what success looks like]

Escalation

Prevention

Long-term actions to prevent recurrence.

• [ ] [Preventive action 1]
• [ ] [Preventive action 2]
• [ ] [Preventive action 3]

Change Log

Writing Effective Runbooks: A Practitioner's Guide

Why Runbooks Matter

A runbook is only as good as its last use under pressure. When an engineer is paged at 3 AM with elevated cortisol and reduced cognitive function, they need clear, unambiguous instructions that eliminate guesswork.

This guide covers how to write runbooks that actually work when it counts.

The Golden Rules

1. Write for the Worst Case

Assume the reader:

• Has never seen this system before
• Is sleep-deprived and stressed
• Cannot reach anyone else for help
• Has only terminal access (no fancy dashboards)

2. Commands Over Prose

Bad:

Check the database replication status and verify it's healthy.

Good:

# Check replication lag in seconds (should be < 5)
psql -h replica.internal -U monitoring -c "SELECT EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp()))::int AS lag_seconds;"

3. Show Expected Output

Every command should include what "normal" looks like. If the responder doesn't know what healthy looks like, they can't distinguish broken from normal.

4. One Decision, One Branch

Never embed multiple decisions in one step. Use clear if/then branching:

Step 3: Check connection count
  → If connections > 90% of max: Go to Mitigation Option A
  → If connections < 50% but queries slow: Go to Step 4
  → If connections normal: This runbook may not apply, check INDEX.md

5. Include Rollback for Every Action

Every mitigation step should include how to undo it. Mitigations that can't be rolled back need explicit warnings.

Runbook Structure

Every runbook in this library follows a consistent 9-section structure:

Writing Tips by Section

Symptoms

• Use checkboxes so responders can quickly confirm which symptoms match
• Include both technical symptoms (metrics, logs) and user-facing symptoms
• List the most common/obvious symptom first
• Include "NOT this runbook if..." to prevent misdiagnosis

Severity Classification

• Use a table mapping conditions to severity levels
• Include customer-impact language, not just technical thresholds
• Match your organization's incident severity definitions

Detection & Alerts

• List the exact alert names and their sources
• Include the threshold that triggers the alert
• Link to the relevant dashboard (even if the link changes, it shows WHERE to look)

Diagnosis Steps

• Number every step sequentially
• Each step: one command, expected output, interpretation
• Include timing — "this command takes ~30 seconds on large clusters"
• If a command requires elevated privileges, say so explicitly

Mitigation

• Always provide at least two options (Plan A and Plan B)
• Label each option's trade-offs: speed vs. safety, temporary vs. permanent
• Include "Risk:" and "Rollback:" for every mitigation
• Prefer the safest option first, fastest option second

Resolution

• This is the full fix, not just the bandaid
• Include steps that may require a maintenance window
• Reference related runbooks if resolution depends on other systems

Verification

• Use checkboxes for multi-step verification
• Include both immediate verification (is it fixed?) and delayed verification (does it stay fixed?)
• Specify how long to monitor before declaring resolved

Escalation

• Include conditions that trigger escalation (time-based and impact-based)
• Provide specific contact methods, not just names
• Include timezone information for distributed teams

Prevention

• Link to relevant Jira/issue templates for follow-up work
• Categorize: quick wins vs. major projects
• Reference industry best practices where applicable

Common Anti-Patterns

Maintenance Schedule

Runbooks are living documents. They rot if not maintained.

Quarterly Review Checklist

• [ ] All commands still work (test in staging)
• [ ] Alert names match current alerting configuration
• [ ] Escalation contacts are still current
• [ ] Dashboard links are still valid
• [ ] Severity thresholds match current SLOs
• [ ] New failure modes have been documented since last review

Trigger-Based Updates

Update a runbook immediately when:

• It's used during an incident and a step is wrong or missing
• Infrastructure changes make a command invalid
• A postmortem identifies a gap in the runbook
• Team members change (escalation contacts)

Version Control

• Keep runbooks in version control (git)
• Use meaningful commit messages: "Update failover runbook: new replica promotion command after PG16 upgrade"
• Tag reviewed versions so you can always diff against the last known-good state

Metrics for Runbook Quality

Track these to measure whether your runbooks are actually useful:

Starting From Scratch

If you're building a runbook library from zero:

1. Mine your incident history — Find your top 10 most common incidents from the last 6 months

2. Write runbooks for the top 5 — Focus on the incidents that happen most frequently

3. Validate immediately — Next time one of those incidents occurs, have the responder follow the runbook literally and note gaps

4. Iterate rapidly — Update the runbook after every use

5. Expand coverage — Add 2-3 new runbooks per sprint until you hit 80% coverage

File Naming Conventions

This library uses the following conventions:

• Lowercase with hyphens: connection-pool-exhaustion.md
• Category folders: database/, compute/, network/, etc.
• Descriptive names that match the symptom or failure mode, not the root cause
• Good: slow-queries.md (symptom-based, easy to find)
• Bad: missing-index.md (root-cause-based, requires diagnosis first)

*Remember: A runbook that's 80% complete and available NOW is infinitely more valuable than a perfect runbook that's "still being drafted."*