Monitoring Strategy Guide

From zero visibility to production-grade observability.

This guide covers the principles, patterns, and pitfalls of building a monitoring

practice that actually works — not just dashboards that look pretty.

Table of Contents

1. The Three Pillars of Observability

2. SLIs, SLOs, and SLAs

3. The RED and USE Methods

4. Alert Design: Avoiding Alert Fatigue

5. Dashboard Design Principles

6. Runbook Template

7. Incident Response Workflow

8. Capacity Planning

The Three Pillars of Observability

Metrics (Prometheus)

• What: Numeric measurements over time (counters, gauges, histograms)
• When to use: Alerting, dashboards, trend analysis, capacity planning
• Example: http_requests_total, node_cpu_seconds_total
• Strength: Low overhead, fast queries, long retention

Logs (Loki)

• What: Timestamped text records of discrete events
• When to use: Debugging specific issues, audit trails, error details
• Example: 2026-03-10T12:34:56Z ERROR failed to connect to database: timeout
• Strength: Rich context, grep-able, human-readable

Traces (future addition)

• What: End-to-end request flow across services
• When to use: Latency analysis, dependency mapping, distributed debugging
• Example: A trace showing request path: API → Auth → DB → Cache → Response
• Strength: Pinpoints where time is spent across service boundaries

Rule of thumb: Use metrics to detect problems, logs to diagnose them, traces to understand service interactions.

SLIs, SLOs, and SLAs

Service Level Indicators (SLIs)

SLIs are the metrics that matter — quantitative measures of your service's behavior from the user's perspective.

Service Level Objectives (SLOs)

SLOs are targets for your SLIs. They define "good enough."

Example SLOs:

Availability:  99.9%  (43.8 min downtime/month)
Latency p95:   < 500ms
Latency p99:   < 1000ms
Error rate:    < 0.1%

Error Budget = 100% - SLO. If your SLO is 99.9%, you have a 0.1% error budget — that's ~43 minutes/month of allowed downtime.

Service Level Agreements (SLAs)

SLAs are contracts with customers. They should always be less strict than your internal SLOs.

Internal SLO:   99.95% availability  (you aim for this)
External SLA:   99.9%  availability  (you promise this)
Gap:            0.05%                (your safety margin)

Implementing SLOs in Prometheus

# Recording rule: SLI — success rate over 30 days
- record: sli:http_availability:ratio_rate30d
  expr: |
    1 - (
      sum(rate(http_requests_total{status=~"5.."}[30d]))
      /
      sum(rate(http_requests_total[30d]))
    )

# Alert: SLO burn rate (multi-window)
- alert: SLOBudgetBurnRateHigh
  expr: |
    (
      sli:http_error_rate:ratio_rate1h > (14.4 * 0.001)
      and
      sli:http_error_rate:ratio_rate5m > (14.4 * 0.001)
    )
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Error budget burn rate is 14.4x — budget exhaustion in ~1 hour"

The RED and USE Methods

RED Method (for request-driven services)

Use RED for: APIs, web servers, microservices, gateways

USE Method (for infrastructure resources)

Use USE for: CPU, memory, disk, network, connection pools

Alert Design: Avoiding Alert Fatigue

The Problem

Most teams start with too many alerts. Within weeks, everyone ignores them. This is alert fatigue — the silent killer of incident response.

Principles

1. Alert on symptoms, not causes

BAD:  CPU > 80%          (cause — might not affect users)
GOOD: Error rate > 1%    (symptom — users are affected)
GOOD: Latency p95 > 1s   (symptom — users notice slowness)

2. Every alert must be actionable

If an engineer gets paged, they must be able to do something about it. If the response is "just wait," it shouldn't be an alert.

3. Use severity levels correctly

4. Tune for durations to avoid flapping

# BAD: fires on every 1-second CPU spike
- alert: HighCPU
  expr: cpu_usage > 90
  for: 0s

# GOOD: only fires if sustained for 10 minutes
- alert: HighCPU
  expr: cpu_usage > 90
  for: 10m

5. Use inhibition to reduce noise

When a host is completely down, you don't need 50 alerts — one "HostDown" alert is enough. AlertManager's inhibit_rules suppress downstream alerts.

Alert Review Process

Run a monthly alert review:

1. List all alerts that fired in the past month

2. For each: Was it actionable? Did someone respond? Was it a real problem?

3. Delete or tune alerts that fail these criteria

4. Add alerts for incidents that weren't caught

Dashboard Design Principles

The 4 Golden Signals Dashboard Pattern

Every service should have a dashboard with these 4 panels:

1. Traffic — requests per second (tells you what's happening)

2. Errors — error rate / count (tells you what's broken)

3. Latency — p50, p95, p99 (tells you how slow it is)

4. Saturation — resource utilization (tells you how full it is)

Layout Best Practices

┌─────────────────────────────────────────────────┐
│  Row 1: Key metrics (stat panels)               │
│  [Request Rate] [Error %] [P95 Latency] [Avail] │
├─────────────────────────────────────────────────┤
│  Row 2: Traffic + Errors (time series)          │
│  [Rate by status code] [Error rate by endpoint]  │
├─────────────────────────────────────────────────┤
│  Row 3: Latency (time series)                   │
│  [Latency percentiles] [Latency by endpoint]     │
├─────────────────────────────────────────────────┤
│  Row 4: Resources (time series)                 │
│  [CPU/Memory] [Connections] [Queue depth]        │
└─────────────────────────────────────────────────┘

Rules

• Top row = quick glance: Use stat panels with thresholds (green/yellow/red)
• Time range default: 1 hour: Long enough to see trends, short enough to be responsive
• Template variables: Add dropdowns for environment, instance, handler
• Consistent colors: Green = good, yellow = degraded, red = bad
• Link dashboards: Node overview → Docker → Application (drill-down)

Runbook Template

Every critical alert should have a runbook. Store these alongside your code.

# Runbook: [AlertName]

## Alert Description
What this alert means and why it matters.

## Impact
- Who is affected (users, internal, specific feature)
- Severity: critical / warning

## Investigation Steps
1. Check the dashboard: [link to Grafana dashboard]
2. Check recent deployments: `git log --oneline -10`
3. Check logs: Grafana → Explore → Loki → `{job="myapp"} |= "error"`
4. Check dependent services: [list]

## Common Causes
| Cause | How to Verify | Fix |
|-------|---------------|-----|
| Memory leak | RSS growing steadily | Restart pod, investigate leak |
| Database slow | Query duration > 5s | Check slow query log |
| Dependency down | Connection refused in logs | Check upstream service |

## Mitigation
1. Immediate: [what to do right now to reduce impact]
2. Short-term: [what to fix within hours]
3. Long-term: [what to fix to prevent recurrence]

## Escalation
- L1: On-call engineer (PagerDuty)
- L2: Team lead
- L3: Platform team

## References
- Architecture doc: [link]
- Previous incidents: [links to postmortems]

Incident Response Workflow

Alert fires
    │
    ├── Acknowledge (PagerDuty / Slack)
    │
    ├── Assess severity (who is affected? how bad?)
    │
    ├── Investigate (dashboards → logs → traces)
    │
    ├── Mitigate (reduce impact now)
    │   ├── Rollback deployment
    │   ├── Scale up resources
    │   ├── Enable circuit breaker
    │   └── Redirect traffic
    │
    ├── Fix (root cause)
    │
    ├── Verify (confirm alert resolves, metrics normalize)
    │
    └── Postmortem (blameless, actionable)
        ├── What happened?
        ├── Timeline of events
        ├── What worked well?
        ├── What could be improved?
        └── Action items (with owners and deadlines)

Capacity Planning

Monitoring Stack Resource Estimates

Scaling Signals

• Prometheus: Scrape duration increasing, memory usage growing, query latency > 10s
• Loki: Ingestion rate hitting limits, query timeouts, chunk store growing fast
• Grafana: Dashboard load time > 5s, concurrent users > 50

When to Scale Out

Datanest Digital | datanest.dev | MIT License

Questions? Email support@datanest.dev

Monitoring Stack Setup

This package runs a complete metrics, alerting, dashboard, and logging pipeline with Docker Compose. Prometheus collects and evaluates metrics, Grafana visualizes them, AlertManager delivers actionable notifications, and Loki stores logs shipped by Promtail.

Prometheus: collection, storage, and alerting

Prometheus uses a pull model: every scrape interval it requests an HTTP metrics endpoint, adds target labels, and writes samples to its local time-series database (TSDB). prometheus/prometheus.yml defines the scrape jobs, loads recording and alert rules, and sends firing alerts to AlertManager. Recording rules precompute expensive, frequently used PromQL; alert rules should describe sustained, actionable conditions rather than momentary spikes.

The included stack already scrapes Prometheus, Node Exporter, cAdvisor, Grafana, AlertManager, and Loki. Add an application by exposing Prometheus-format metrics and using a hostname reachable from the Prometheus container:

scrape_configs:
  - job_name: orders-api
    metrics_path: /metrics
    scrape_interval: 15s
    static_configs:
      - targets: ["orders-api:8080"]
        labels:
          environment: production
          team: checkout

Reload with curl -X POST http://localhost:9090/-/reload, then check Status → Targets. Instrument request count, error count, and duration histograms. Keep labels bounded: route templates, methods, and status classes are useful; customer IDs, request IDs, and full URLs create costly cardinality.

Node Exporter and application metrics

Node Exporter reads host CPU, memory, filesystem, disk, and network counters through read-only host mounts. Its data powers node-overview.json and infrastructure alerts. cAdvisor supplies complementary per-container resource metrics. These exporters reveal resource pressure, but application metrics reveal customer impact. Use the RED signals—request rate, error rate, and duration—for APIs, and pair them with Node Exporter’s USE signals—utilization, saturation, and errors—during diagnosis.

Grafana provisioning

Grafana provisioning makes environments repeatable. On startup, grafana/provisioning/datasources.yaml registers Prometheus and Loki through internal Compose DNS, while dashboards.yaml loads the three supplied dashboard JSON files. A minimal datasource definition is:

apiVersion: 1
datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      timeInterval: 15s
  - name: Loki
    uid: loki
    type: loki
    access: proxy
    url: http://loki:3100

Start with the node, Docker, and application dashboards, then align panel queries with your metric names and labels. If you edit a provisioned dashboard in the UI, export it back to its JSON file so the change survives replacement or redeployment.

AlertManager: routing, silencing, and inhibition

AlertManager groups related alerts, routes them by labels, deduplicates repeats, and sends resolved notifications. The supplied policy sends critical alerts to PagerDuty and warnings or informational alerts to Slack. A compact equivalent is:

route:
  receiver: slack-warnings
  group_by: [alertname, job]
  group_wait: 30s
  routes:
    - matchers: ['severity="critical"']
      receiver: pagerduty-critical
      repeat_interval: 1h
inhibit_rules:
  - source_matchers: ['severity="critical"']
    target_matchers: ['severity="warning"']
    equal: [alertname, instance]
receivers:
  - name: slack-warnings
    slack_configs:
      - channel: "#alerts-warning"
        send_resolved: true
  - name: pagerduty-critical
    pagerduty_configs:
      - routing_key: YOUR_ROUTING_KEY
        send_resolved: true

Replace placeholders through a secret-management or deployment-templating step; never commit real receiver credentials. Silences are temporary runtime objects created in the AlertManager UI or with amtool, using label matchers, an owner, a reason, and an expiry. Use them for planned maintenance. Inhibition is configuration-driven: it automatically suppresses secondary warnings when a causally broader critical alert, such as HostDown, is active.

Loki and production operation

Promtail tails Docker, system, and authentication logs, applies low-cardinality labels, and pushes batches to Loki. Loki indexes labels rather than full log text, keeping storage economical. In Grafana Explore, query a stream such as {job="docker"} |= "error" and correlate its time range with a Prometheus alert. Keep request IDs in log bodies, not labels.

The bundle is a strong single-host baseline, not automatic high availability. Set Prometheus time- and size-based retention to fit disk capacity, monitor the metrics and Loki volumes, and back up Grafana state and configuration. For multiple sites, federate selected aggregate series to a central Prometheus or use remote write for durable long-term storage. For HA, run duplicate Prometheus servers with identical targets, cluster multiple AlertManagers for deduplication, and place Grafana behind a load balancer with a shared database. Loki HA requires object storage and replicated distributed components. Finally, protect all administrative endpoints with private networking, TLS, authentication, and least-privilege host mounts.