Monitoring Strategy Guide

From zero visibility to production-grade observability.

This guide covers the principles, patterns, and pitfalls of building a monitoring

practice that actually works — not just dashboards that look pretty.

Table of Contents

1. The Three Pillars of Observability

2. SLIs, SLOs, and SLAs

3. The RED and USE Methods

4. Alert Design: Avoiding Alert Fatigue

5. Dashboard Design Principles

6. Runbook Template

7. Incident Response Workflow

8. Capacity Planning

The Three Pillars of Observability

Metrics (Prometheus)

• What: Numeric measurements over time (counters, gauges, histograms)
• When to use: Alerting, dashboards, trend analysis, capacity planning
• Example: http_requests_total, node_cpu_seconds_total
• Strength: Low overhead, fast queries, long retention

Logs (Loki)

• What: Timestamped text records of discrete events
• When to use: Debugging specific issues, audit trails, error details
• Example: 2026-03-10T12:34:56Z ERROR failed to connect to database: timeout
• Strength: Rich context, grep-able, human-readable

Traces (future addition)

• What: End-to-end request flow across services
• When to use: Latency analysis, dependency mapping, distributed debugging
• Example: A trace showing request path: API → Auth → DB → Cache → Response
• Strength: Pinpoints where time is spent across service boundaries

Rule of thumb: Use metrics to detect problems, logs to diagnose them, traces to understand service interactions.

SLIs, SLOs, and SLAs

Service Level Indicators (SLIs)

SLIs are the metrics that matter — quantitative measures of your service's behavior from the user's perspective.

Service Level Objectives (SLOs)

SLOs are targets for your SLIs. They define "good enough."

Example SLOs:

Availability: 99.9% (43.8 min downtime/month)
Latency p95:  < 500ms
Latency p99:  < 1000ms
Error rate:  < 0.1%

Error Budget = 100% - SLO. If your SLO is 99.9%, you have a 0.1% error budget — that's ~43 minutes/month of allowed downtime.

Service Level Agreements (SLAs)

SLAs are contracts with customers. They should always be less strict than your internal SLOs.

Internal SLO:  99.95% availability (you aim for this)
External SLA:  99.9% availability (you promise this)
Gap:      0.05%        (your safety margin)

Implementing SLOs in Prometheus

# Recording rule: SLI — success rate over 30 days
- record: sli:http_availability:ratio_rate30d
 expr: |
  1 - (
   sum(rate(http_requests_total{status=~"5.."}[30d]))
   /
   sum(rate(http_requests_total[30d]))
  )

# Alert: SLO burn rate (multi-window)
- alert: SLOBudgetBurnRateHigh
 expr: |
  (
   sli:http_error_rate:ratio_rate1h > (14.4 * 0.001)
   and
   sli:http_error_rate:ratio_rate5m > (14.4 * 0.001)
  )
 for: 2m
 labels:
  severity: critical
 annotations:
  summary: "Error budget burn rate is 14.4x — budget exhaustion in ~1 hour"

The RED and USE Methods

RED Method (for request-driven services)

Use RED for: APIs, web servers, microservices, gateways

USE Method (for infrastructure resources)

Use USE for: CPU, memory, disk, network, connection pools

Alert Design: Avoiding Alert Fatigue

The Problem

Most teams start with too many alerts. Within weeks, everyone ignores them. This is alert fatigue — the silent killer of incident response.

Principles

1. Alert on symptoms, not causes

BAD: CPU > 80%     (cause — might not affect users)
GOOD: Error rate > 1%  (symptom — users are affected)
GOOD: Latency p95 > 1s  (symptom — users notice slowness)

2. Every alert must be actionable

If an engineer gets paged, they must be able to do something about it. If the response is "just wait," it shouldn't be an alert.

3. Use severity levels correctly

4. Tune for durations to avoid flapping

# BAD: fires on every 1-second CPU spike
- alert: HighCPU
 expr: cpu_usage > 90
 for: 0s

# GOOD: only fires if sustained for 10 minutes
- alert: HighCPU
 expr: cpu_usage > 90
 for: 10m

5. Use inhibition to reduce noise

When a host is completely down, you don't need 50 alerts — one "HostDown" alert is enough. AlertManager's inhibit_rules suppress downstream alerts.

Alert Review Process

Run a monthly alert review:

1. List all alerts that fired in the past month

2. For each: Was it actionable? Did someone respond? Was it a real problem?

3. Delete or tune alerts that fail these criteria

4. Add alerts for incidents that weren't caught

Dashboard Design Principles

The 4 Golden Signals Dashboard Pattern

Every service should have a dashboard with these 4 panels:

1. Traffic — requests per second (tells you what's happening)

2. Errors — error rate / count (tells you what's broken)

3. Latency — p50, p95, p99 (tells you how slow it is)

4. Saturation — resource utilization (tells you how full it is)

Layout Best Practices

┌─────────────────────────────────────────────────┐
│ Row 1: Key metrics (stat panels)        │
│ [Request Rate] [Error %] [P95 Latency] [Avail] │
├─────────────────────────────────────────────────┤
│ Row 2: Traffic + Errors (time series)     │
│ [Rate by status code] [Error rate by endpoint] │
├─────────────────────────────────────────────────┤
│ Row 3: Latency (time series)          │
│ [Latency percentiles] [Latency by endpoint]   │
├─────────────────────────────────────────────────┤
│ Row 4: Resources (time series)         │
│ [CPU/Memory] [Connections] [Queue depth]    │
└─────────────────────────────────────────────────┘

Rules

• Top row = quick glance: Use stat panels with thresholds (green/yellow/red)
• Time range default: 1 hour: Long enough to see trends, short enough to be responsive
• Template variables: Add dropdowns for environment, instance, handler
• Consistent colors: Green = good, yellow = degraded, red = bad
• Link dashboards: Node overview → Docker → Application (drill-down)

Runbook Template

Every critical alert should have a runbook. Store these alongside your code.

# Runbook: [AlertName]

## Alert Description
What this alert means and why it matters.

## Impact
- Who is affected (users, internal, specific feature)
- Severity: critical / warning

## Investigation Steps
1. Check the dashboard: [link to Grafana dashboard]
2. Check recent deployments: `git log --oneline -10`
3. Check logs: Grafana → Explore → Loki → `{job="myapp"} |= "error"`
4. Check dependent services: [list]

## Common Causes
| Cause | How to Verify | Fix |
|-------|---------------|-----|
| Memory leak | RSS growing steadily | Restart pod, investigate leak |
| Database slow | Query duration > 5s | Check slow query log |
| Dependency down | Connection refused in logs | Check upstream service |

## Mitigation
1. Immediate: [what to do right now to reduce impact]
2. Short-term: [what to fix within hours]
3. Long-term: [what to fix to prevent recurrence]

## Escalation
- L1: On-call engineer (PagerDuty)
- L2: Team lead
- L3: Platform team

## References
- Architecture doc: [link]
- Previous incidents: [links to postmortems]

Incident Response Workflow

Alert fires
  │
  ├── Acknowledge (PagerDuty / Slack)
  │
  ├── Assess severity (who is affected? how bad?)
  │
  ├── Investigate (dashboards → logs → traces)
  │
  ├── Mitigate (reduce impact now)
  │  ├── Rollback deployment
  │  ├── Scale up resources
  │  ├── Enable circuit breaker
  │  └── Redirect traffic
  │
  ├── Fix (root cause)
  │
  ├── Verify (confirm alert resolves, metrics normalize)
  │
  └── Postmortem (blameless, actionable)
    ├── What happened?
    ├── Timeline of events
    ├── What worked well?
    ├── What could be improved?
    └── Action items (with owners and deadlines)

Capacity Planning

Monitoring Stack Resource Estimates

Scaling Signals

• Prometheus: Scrape duration increasing, memory usage growing, query latency > 10s
• Loki: Ingestion rate hitting limits, query timeouts, chunk store growing fast
• Grafana: Dashboard load time > 5s, concurrent users > 50

When to Scale Out

Datanest Digital | datanest.dev | MIT License

Questions? Email support@datanest.dev

Monitoring Stack Setup

Complete observability stack with Prometheus, Grafana, Loki, and AlertManager.

One docker compose up to full production monitoring. Stop flying blind.