Alerting Setup Guide

This guide covers how to turn the metrics from the ML Monitoring Suite into

actionable alerts using the bundled rule engine (ml_monitoring.alerting). It

explains rule design, how to suppress flapping, how to route by severity, and how

to wire alerts into the destination of your choice — all with zero third-party

dependencies.

How the engine works

You register Rule objects with an AlertManager, then call

manager.evaluate(metrics, timestamp=...) once per cycle with a flat

{metric_name: value} mapping. The manager is stateful: it remembers which

alerts are currently firing, so each rule fires exactly once when it starts

breaching and emits a matching resolved event when it recovers. You get incident

*transitions*, not a wall of duplicate notifications.

from ml_monitoring import AlertManager, ThresholdRule, AnomalyRule, Severity
from ml_monitoring import ConsoleSink, JSONLinesSink

manager = AlertManager(
    sinks=[ConsoleSink(), JSONLinesSink("/var/log/ml-alerts.jsonl")],
    default_labels={"model": "churn_classifier", "env": "production"},
)

Each evaluation returns the list of new firing/resolved Alert events, and

manager.active_alerts() returns everything currently firing.

Building the metric snapshot

The engine evaluates a plain dict, which you assemble from the monitors each cycle.

Keep the keys stable — they are what rules match on.

latency = monitor.latency_window().stats()
drift_results = drift.check({"tenure_months": recent_batch})

snapshot = {
    "error_rate": monitor.error_rate(),
    "latency_p95_ms": latency.p95,
    "throughput_per_min": monitor.throughput_per_minute(),
    "tenure_drift_psi": drift_results["tenure_months"].drift_score,
    "dq_failures": len(report.failures),
}
events = manager.evaluate(snapshot, timestamp=time.time())

Threshold rules

ThresholdRule fires when metric threshold holds. Operators are

">", ">=", "<", "<=", "==", "!=". This is the right tool when you have a

known SLO.

manager.add_rule(ThresholdRule(
    "high_error_rate", "error_rate", ">", 0.05,
    severity=Severity.CRITICAL,
    for_seconds=300,          # must breach continuously for 5 minutes
))

Suppressing flapping with for_seconds

A metric that hovers around its threshold will otherwise fire and resolve

repeatedly. for_seconds is a dwell time, exactly like a Prometheus for: clause:

the rule must breach *continuously* for that long before the manager fires it. Use

it on any noisy signal (latency, error rate). A for_seconds=0 rule (the default)

fires immediately — appropriate for hard, must-never-happen conditions like a

data-quality error.

Because the dwell is measured against the timestamp you pass in, it is fully

deterministic and testable: feed timestamps 0, 5, 10 and a rule with

for_seconds=10 fires precisely on the third evaluation.

Anomaly rules

When a metric has no obvious fixed threshold (a per-model throughput, a feature

mean that varies by season), use AnomalyRule. It maintains its own rolling

baseline and fires when a value is more than z_threshold standard deviations from

the recent mean. It only starts alerting once min_samples of baseline have

accumulated, so it never fires on cold start.

manager.add_rule(AnomalyRule(
    "throughput_anomaly", "throughput_per_min",
    z_threshold=4.0,          # 4 sigma — conservative, few false positives
    window_size=200,          # baseline = last 200 observations
    min_samples=30,           # warm up before alerting
    severity=Severity.WARNING,
))

The baseline adapts over time, so a slow, legitimate ramp in traffic will not keep

alerting — only abrupt deviations do.

A complete starter rule set

These rules cover the four golden signals plus data quality. They are a good

production starting point for an online classifier.

manager.add_rule(ThresholdRule("high_error_rate", "error_rate", ">", 0.05,
                                severity=Severity.CRITICAL, for_seconds=300))
manager.add_rule(ThresholdRule("slow_p95", "latency_p95_ms", ">", 250.0,
                                severity=Severity.WARNING, for_seconds=300))
manager.add_rule(ThresholdRule("feature_drift", "tenure_drift_psi", ">=", 0.25,
                                severity=Severity.WARNING))
manager.add_rule(ThresholdRule("dq_failure", "dq_failures", ">", 0,
                                severity=Severity.CRITICAL))   # any failure pages
manager.add_rule(AnomalyRule("throughput_anomaly", "throughput_per_min",
                             z_threshold=4.0, min_samples=30))

Severity and routing

Severity has three levels — INFO, WARNING, CRITICAL. Map them to your

on-call policy:

• CRITICAL — wake someone up. Reserve for customer-visible breakage: high error

rate, data-quality errors that would feed garbage to the model.

• WARNING — create a ticket / Slack message. Drift, slow latency, throughput

anomalies — things to fix this week, not tonight.

• INFO — record only. Useful for audit trails and post-incident timelines.

The severity is attached to every Alert, so your sink can route on it.

Sinks: where alerts go

Sinks are pluggable; a failing sink never blocks the others (the manager catches and

logs sink exceptions). Built-in sinks:

A webhook bridge is just a CallableSink:

import json, urllib.request

def post_to_webhook(alert):
    payload = json.dumps(alert.as_dict()).encode("utf-8")
    req = urllib.request.Request(
        "https://hooks.example.com/services/your-webhook",
        data=payload, headers={"Content-Type": "application/json"},
    )
    urllib.request.urlopen(req, timeout=5)

manager.add_sink(CallableSink(post_to_webhook))

Only fire the webhook for the severities you care about by filtering inside the

callable (if alert.severity is Severity.CRITICAL: ...).

Avoiding alert fatigue

Alert fatigue is the failure mode that makes monitoring worthless. Defend against

it deliberately:

1. Use for_seconds on every noisy signal. Latency and error rate should never

page on a single spiky scrape.

2. Page only on CRITICAL. Everything else is a ticket. If a WARNING pages you at

3am, downgrade it or add dwell time.

3. Alert on symptoms, not causes. Page on the error rate the customer feels, not

on every internal counter that might contribute to it.

4. Make drift a ticket, not a page. Drift is a slow process; a same-day ticket is

the right urgency. Only the significant band (PSI ≥ 0.25) is worth a rule.

5. Resolve loudly. The resolved events the manager emits are as valuable as the

firing ones — they tell on-call the incident is over without a manual all-clear.

Incident runbook (paste into your own docs)

When high_error_rate fires:

1. Check throughput — did errors rise, or did successful predictions fall? A

throughput collapse points upstream (caller/feature service), not at the model.

2. Inspect the latest data-quality report. A spike in ml_data_quality_failures_total

at the same time means malformed features, not a model fault.

3. Check ml_drift_score. Significant drift plus rising errors is the classic

"the world changed" signal — start a retrain, and consider rolling back to the

previous model version in the meantime.

4. If latency p95 also breached, suspect resource saturation (CPU/GPU, batch size,

a slow dependency) rather than the model weights.

5. Confirm recovery via the matching resolved alert before closing the incident.

Production ML Monitoring Guide

This guide explains how to wire the ML Monitoring Suite into a serving process and

how to read what it tells you. It is opinionated: it describes the signals that

actually predict model incidents, the thresholds that work in practice, and the

failure modes the suite is designed to catch. Every API named here is real and

lives in src/ml_monitoring/.

The four golden signals for a served model

Classic SRE monitoring tracks latency, traffic, errors and saturation. A model in

production needs four analogous signals, and this suite produces all of them from a

single stream of record_prediction() calls:

1. Volume / throughput — how many predictions you serve. A sudden drop usually

means an upstream caller broke, not the model. Exposed as ml_predictions_total

(counter) and ml_throughput_predictions_per_minute (gauge).

2. Latency — how long inference takes, as percentiles (p50/p90/p95/p99). Tail

latency, not the mean, is what your callers feel. Exposed as

ml_prediction_latency_milliseconds{quantile="..."}.

3. Errors — the fraction of predictions that failed (an exception, a timeout, a

guard-rail rejection). Exposed as ml_prediction_errors_total and

ml_prediction_error_rate.

4. Drift — how far the live input/output distribution has moved from the data

the model was trained on. This is the *unique* ML signal: a model can be fast,

error-free and still quietly wrong because the world changed. Exposed as

ml_drift_score and ml_drift_detected.

Data quality sits underneath all four: if the features are malformed, every other

signal is meaningless. The suite treats it as a first-class check

(ml_data_quality_failures_total, ml_data_quality_check_status).

Minimal wiring

from ml_monitoring import (
    PredictionMonitor, DriftMonitor, CollectorRegistry,
    PrometheusExporter, MetricsServer,
)

monitor = PredictionMonitor("churn_classifier", version="2.3.0", window_seconds=300)
drift = DriftMonitor(model="churn_classifier", psi_threshold=0.2)
drift.fit_numeric("tenure_months", reference_tenure_values)  # fit once, offline

registry = CollectorRegistry()
registry.register(monitor)
registry.register(drift)

# Expose /metrics for Prometheus to scrape (background daemon thread).
server = MetricsServer(PrometheusExporter(registry), host="0.0.0.0", port=9188).start()

In your prediction path, record one observation per request:

def predict(features):
    started = time.perf_counter()
    try:
        score = model.predict_proba(features)
        label = "churn" if score >= 0.5 else "retain"
        monitor.record_prediction(
            value=score,
            latency_ms=(time.perf_counter() - started) * 1000.0,
            label=label,
        )
        return score
    except Exception:
        monitor.record_error(latency_ms=(time.perf_counter() - started) * 1000.0)
        raise

record_prediction() is cheap and thread-safe, so it is safe to call on the hot

path of a multi-threaded server.

Rolling windows, not lifetime averages

Latency, throughput and the output-value distribution are all computed over a

rolling time window (window_seconds, default 300s) backed by

ml_monitoring.store.RollingWindow. This matters: a lifetime p95 hides a

regression that started an hour ago behind months of healthy history. The window

evicts by *event time* (the largest timestamp it has seen), which also makes the

behaviour deterministic and unit-testable — pass an explicit timestamp= to

replay historical data and get identical results.

Counters (predictions_total, errors_total) are intentionally lifetime

totals, because Prometheus computes rates from counters itself

(rate(ml_predictions_total[5m])). Do not reset them in production.

Drift detection methodology

The suite ships two complementary tests so you can trigger on magnitude and on

statistical significance:

PSI is the workhorse. Use the conventional interpretation bands, which the suite

encodes as the severity field on every DriftResult:

• PSI < 0.10 — none. No meaningful drift; do nothing.
• 0.10 ≤ PSI < 0.25 — moderate. Investigate; consider a refresh soon.
• PSI ≥ 0.25 — significant. The input distribution has materially shifted;

schedule a retrain and check for upstream data changes.

Fit the reference once on a representative slice of training (or recent, known-good

production) data, then score each live batch:

results = drift.check({"tenure_months": recent_tenure_batch})
if results["tenure_months"].drifted:
    print(results["tenure_months"].as_dict())  # psi, ks_statistic, ks_pvalue, severity

Monitor the model's output distribution too, not just its inputs. A shift in

ml_prediction_value (the score distribution) is often the earliest visible sign of

concept drift, because it moves before you have ground-truth labels to measure

accuracy with.

Data-quality gating

Run a DataQualitySuite against each feature batch *before* it reaches the model.

The expectations cover the corruption that breaks pipelines in practice: missing

columns, nulls in non-nullable fields, out-of-range or wrong-typed values, unknown

categorical levels, and collapsed or exploded cardinality.

report = feature_suite.validate(batch)        # batch is a list[dict]
if not report.success:                          # only error-severity checks fail it
    log.error("feature batch rejected: %s", report.summary())
    raise FeatureQualityError(report.error_failures)

Use severity="warning" for soft expectations (e.g. "mean spend looks low") so they

are reported on the dashboard without blocking the scoring job, and tighten them to

"error" once you trust them.

Exposition and dashboards

The exporters are dependency-free. PrometheusExporter.render() returns the text

exposition format that Prometheus scrapes; JSONExporter.render() returns the same

families as JSON for log shipping or a bespoke UI. Point Prometheus at the

MetricsServer endpoint:

# prometheus.yml
scrape_configs:
  - job_name: ml-monitoring
    scrape_interval: 30s
    static_configs:
      - targets: ["model-server.your-org.internal:9188"]

Then import dashboards/grafana-ml-monitoring.json into Grafana (9.x or newer) and

select your Prometheus data source when prompted. The dashboard has a model

template variable, so one dashboard serves every model that reports to the same

Prometheus.

Recommended SLOs and starting thresholds

These are sensible defaults for an online classifier; tune them to your product.

See guides/alerting-setup.md for how to turn these into rules with the bundled

alert engine.

Troubleshooting

• throughput_per_minute reads 0 with traffic flowing. The rate needs at least

two observations spanning a non-zero interval; it warms up within one window.

• PSI looks huge (> 5) on the first batch. Your live batch fell entirely outside

the reference's bins. That is real, extreme drift — but double-check you fitted the

reference on the right feature and units.

• KS p-value is ~0 but PSI is small. KS is extremely sensitive on large samples;

a tiny, harmless shift can be "statistically significant". Trust PSI's magnitude

bands for action and treat KS as a secondary, sensitivity-tunable trigger.

• Latency percentiles look too low. Confirm you pass latency_ms in

milliseconds (not seconds) to record_prediction().