← Back to all products

ML Monitoring Suite

$39

Model monitoring dashboards, alert configurations, data quality checks, and performance tracking.

📁 17 files🏷 v1.0.0
JSONMarkdownPythonGrafanaPrometheus

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 17 files

ml-monitoring-suite/ ├── LICENSE ├── README.md ├── dashboards/ │ └── grafana-ml-monitoring.json ├── examples/ │ ├── data_quality_check.py │ └── monitor_predictions.py ├── guides/ │ ├── alerting-setup.md │ └── monitoring-guide.md ├── src/ │ └── ml_monitoring/ │ ├── __init__.py │ ├── alerting.py │ ├── data_quality.py │ ├── drift_monitor.py │ ├── exporters.py │ ├── metrics.py │ └── store.py └── tests/ ├── test_data_quality.py └── test_metrics.py

📖 Documentation Preview README excerpt

ML Monitoring Suite

Production model monitoring with zero third-party dependencies. A small,

batteries-included Python package that keeps a served model healthy: prediction /

latency / throughput metrics, data-quality expectations, online drift detection

(PSI / KS / Jensen-Shannon), a stateful alert engine, Prometheus + JSON exporters,

and a ready-to-import Grafana dashboard.

Everything is pure Python standard library (3.9+), so it imports and runs in any

environment your model already runs in — no pip install required to use it.

Table of Contents

  • [Why this exists](#why-this-exists)
  • [What's included](#whats-included)
  • [Requirements](#requirements)
  • [Installation](#installation)
  • [Project structure](#project-structure)
  • [Quick start](#quick-start)
  • [The modules](#the-modules)
  • [Metrics reference](#metrics-reference)
  • [Data-quality checks](#data-quality-checks)
  • [Drift detection](#drift-detection)
  • [Alerting](#alerting)
  • [Grafana dashboard](#grafana-dashboard)
  • [Running the examples](#running-the-examples)
  • [Running the tests](#running-the-tests)
  • [FAQ](#faq)
  • [Support](#support)
  • [License](#license)

Why this exists

A model in production needs four signals: volume/throughput, latency,

errors, and drift — plus data quality underneath them all. Most teams

bolt this together from prometheus_client, great_expectations, scipy.stats,

and a pile of glue code, then fight the dependency tree every time their serving

image is rebuilt.

This suite gives you all five signals from a single stream of

record_prediction() calls, implemented from scratch on the standard library so it

can drop into a constrained serving environment (a locked-down base image, a

lambda, an air-gapped box) without adding a single dependency.

What's included

  • Prediction metrics (metrics.py) — a thread-safe PredictionMonitor that

derives volume, error rate, latency percentiles (p50/p90/p95/p99), throughput,

output-value distribution, and per-class counts from one call per request.

  • Rolling storage (store.py) — time- and size-bounded RollingWindow /

MetricStore primitives with deterministic, event-time eviction (fully testable

by passing explicit timestamps).

  • Data-quality expectations (data_quality.py) — a pandas-free, 8-expectation

framework (DataQualitySuite) for schema, null, range, type, set-membership,

cardinality and table-shape checks, with error/warning severities.

  • Drift detection (drift_monitor.py) — online Population Stability Index,

two-sample Kolmogorov-Smirnov (with an asymptotic p-value), and Jensen-Shannon

divergence, wrapped in numeric/categorical detectors and a DriftMonitor.

  • Alerting (alerting.py) — a stateful AlertManager with ThresholdRule

(with Prometheus-style for_seconds dwell) and a self-tuning z-score

... continues with setup instructions, usage examples, and more.

📄 Code Sample .py preview

src/ml_monitoring/alerting.py """ Rule engine and pluggable alert sinks for model monitoring. You register :class:`Rule` objects with an :class:`AlertManager`, then feed the manager a flat ``{metric_name: value}`` mapping each evaluation cycle. The manager tracks alert *state* across cycles, so it fires an alert once when a rule starts breaching and emits a matching *resolved* event when it recovers -- no duplicate spam, and you always know when an incident clears. Two rule families are provided: * :class:`ThresholdRule` -- the workhorse: ``metric <op> threshold`` with an optional ``for_seconds`` dwell time (the breach must persist before firing, exactly like a Prometheus ``for:`` clause), which suppresses flapping. * :class:`AnomalyRule` -- a self-tuning z-score detector that learns a rolling mean/standard deviation and fires when a value is more than ``z_threshold`` standard deviations away. Good for metrics with no obvious fixed threshold. Alerts are dispatched to any number of :class:`AlertSink` implementations (:class:`ConsoleSink`, :class:`JSONLinesSink`, :class:`InMemorySink`, :class:`CallableSink`); a failing sink never blocks the others. """ from __future__ import annotations import json import logging import time from dataclasses import dataclass, field from enum import Enum from typing import Callable, Optional from .store import RollingWindow __all__ = [ "Severity", "Alert", "Rule", "ThresholdRule", "AnomalyRule", "AlertSink", "ConsoleSink", "InMemorySink", "JSONLinesSink", "CallableSink", "AlertManager", ] _LOGGER = logging.getLogger("ml_monitoring.alerting") # ... 346 more lines ...
Buy Now — $39 Back to Products