Error Budget Policy & Multi-Burn-Rate Alerting Guide

What is an Error Budget?

An error budget quantifies the acceptable amount of unreliability for a service. It's derived directly from the SLO target:

Error Budget = 1 - SLO Target

For a 99.9% availability SLO over 30 days:

• Error budget = 0.1% of all requests may fail
• In time terms: ~43.2 minutes of total downtime allowed
• In event terms: 1 in every 1,000 requests may fail

The error budget is not a target to hit — it's a budget to spend. Like any budget, it enables informed tradeoffs between velocity (shipping features) and reliability (keeping things stable).

The Error Budget Policy

An error budget policy is the organizational agreement on what happens as budget depletes. Without a policy, the error budget is just a number. With a policy, it becomes an automated decision framework.

Recommended Thresholds

Policy Enforcement

The policy should be:

1. Written down — documented in a team charter or handbook page

2. Automated — CI/CD gates that check budget before allowing deploys

3. Escalating — each threshold triggers progressively stronger actions

4. Time-bounded — specify how long after recovery before lifting restrictions

Example Policy Document

TEAM: Platform Engineering
EFFECTIVE: 2024-01-01

When the error budget for any Tier 1 or Tier 2 SLO drops below:

50%: SRE must approve all deployments to affected service.
     Deployments require a rollback plan and canary phase.

25%: Only bug fixes and reliability improvements may be deployed.
     Feature development continues but is not released.

10%: Complete deploy freeze. Only emergency patches with SRE approval.
     Team redirects 50% of engineering time to reliability.

 0%: Mandatory post-incident review within 48 hours.
     Feature freeze extends until budget recovers to 25%.
     Executive stakeholders notified.

Budget recovery: After budget climbs back above the threshold,
maintain the restriction for 24 hours before relaxing.

Multi-Window Multi-Burn-Rate Alerting

The Problem with Simple Threshold Alerts

A naive approach to SLO alerting is: "alert when SLI drops below target." This has serious problems:

1. Too slow for fast burns — A complete outage will exhaust a 30-day budget in 43 minutes. A simple "SLI < 99.9%" alert evaluated hourly won't catch it in time.

2. Too noisy for slow burns — A slight degradation that barely dips below target will fire and clear repeatedly, causing alert fatigue without actionable signal.

3. No severity differentiation — A 1-minute blip and a sustained 6-hour degradation both fire the same alert.

The Solution: Burn Rate

Burn rate measures how fast you're consuming error budget relative to the steady-state rate:

Burn Rate = Observed Error Rate / Allowed Error Rate

Where Allowed Error Rate = 1 - SLO Target.

A burn rate of 1.0 means you're consuming budget at exactly the rate that would exhaust it at window end — sustainable but with zero margin.

A burn rate of 14.4 means you'll exhaust a 30-day budget in about 50 hours (30 days / 14.4 ≈ 2.08 days, or ~50 hours).

Multi-Window Strategy

Each alert tier uses two windows:

• Long window: Gives confidence the burn is real (not a blip)
• Short window: Enables auto-resolution when the issue is fixed

An alert fires only when both windows exceed the burn rate threshold simultaneously.

Alert fires when:
  error_rate(long_window)  > burn_rate * allowed_error_rate
  AND
  error_rate(short_window) > burn_rate * allowed_error_rate

Standard Alert Tiers (30-day SLO)

How the Numbers are Derived

The burn rate for each tier is calculated from: "How much budget should be consumed before we alert?"

Burn Rate = Budget Consumption Target / Window Fraction

Where:
  Window Fraction = Long Window / Total SLO Window

For Tier 1 (2% budget consumed, 1h window, 30-day SLO):

Window Fraction = 1 hour / 720 hours = 0.00139
Burn Rate = 0.02 / 0.00139 = 14.4

This means: "Alert when error rate is 14.4x the sustainable rate, sustained for 1 hour."

Why Each Tier Exists

Tier 1 (14.4x, 1h/5min) — "The building is on fire"

• Catches complete outages and severe degradation
• Pages immediately because 2% of monthly budget is gone in 1 hour
• 5-minute short window means it auto-resolves quickly after mitigation

Tier 2 (6.0x, 6h/30min) — "Significant sustained degradation"

• Catches issues that aren't total outages but are burning fast
• 6-hour window filters out short spikes that self-heal
• Still pages because 5% budget loss in 6 hours needs human attention

Tier 3 (3.0x, 1d/2h) — "Slow burn needing investigation"

• Catches subtle degradations (elevated timeout rates, partial failures)
• Creates a ticket — doesn't wake anyone up at 3 AM
• 2-hour short window prevents alert flapping

Tier 4 (1.0x, 3d/6h) — "On track to miss the SLO"

• Catches the "death by a thousand cuts" scenario
• Consuming budget at exactly the unsustainable rate
• Creates a ticket for proactive investigation before breach

Implementation Example (Prometheus)

# Tier 1: Critical burn (pages immediately)
- alert: SLOBurnRate_Critical
  expr: |
    (
      sum(rate(http_requests_total{code=~"5.."}[1h]))
      / sum(rate(http_requests_total[1h]))
      > 14.4 * 0.001  # burn_rate * allowed_error_rate
    )
    and
    (
      sum(rate(http_requests_total{code=~"5.."}[5m]))
      / sum(rate(http_requests_total[5m]))
      > 14.4 * 0.001
    )
  labels:
    severity: page
  annotations:
    summary: "SLO burn rate critical - budget exhaustion imminent"

Tuning Your Alert Tiers

The standard tiers work well for most 30-day SLOs, but you may need to tune:

For 7-day SLOs (aggressive): Derive new tiers using the same formula with a smaller window denominator. Budget consumption targets stay the same.

For batch/pipeline SLOs: Consider only Tier 3 and 4 (ticket severity). Batch processing rarely needs a page-level alert.

For ultra-high-traffic services: You can tighten the short windows because you get statistical significance faster with more data points.

For low-traffic services: Widen windows to avoid alerting on statistically insignificant error counts. A single failed request out of 10 is 10% error rate but may not warrant attention.

Anti-Patterns to Avoid

1. Alerting on raw error counts — "Alert if > 100 errors/min" breaks when traffic scales up or down.

2. Using only one window — Single-window alerts either fire too late (long window) or flap constantly (short window).

3. Same severity for all tiers — If everything pages, nothing is urgent. Reserve pages for Tier 1 and 2.

4. No short window — Without the reset mechanism, alerts stay firing long after the issue is resolved, training operators to ignore them.

5. Alerting on instantaneous values — A 1-second spike to 50% error rate is normal variance, not an incident.

Integrating with Incident Management

When a burn rate alert fires, the response should be:

1. Acknowledge — Prevents escalation and signals team awareness

2. Assess — Is this a real incident or a measurement artifact?

3. Mitigate — Rollback, reroute traffic, scale up, toggle feature flags

4. Communicate — Update status page if user-facing

5. Resolve — Confirm short window clears (alert auto-resolves)

6. Review — Correlate with deployment timeline, update runbooks

Mapping to Incident Severity

Further Reading

• Google SRE Book, Chapter 4: Service Level Objectives
• Google SRE Workbook, Chapter 5: Alerting on SLOs
• The Art of SLOs (Google Cloud whitepaper)
• Implementing SLOs (O'Reilly)