← Back to all products

ML Monitoring Suite

$39

Model monitoring dashboards, alert configurations, data quality checks, and performance tracking.

📁 17 files🏷 v1.0.0
JSONMarkdownPythonGrafanaPrometheus

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 17 files

ml-monitoring-suite/ ├── LICENSE ├── README.md ├── dashboards/ │ └── grafana-ml-monitoring.json ├── examples/ │ ├── data_quality_check.py │ └── monitor_predictions.py ├── guides/ │ ├── alerting-setup.md │ └── monitoring-guide.md ├── src/ │ └── ml_monitoring/ │ ├── __init__.py │ ├── alerting.py │ ├── data_quality.py │ ├── drift_monitor.py │ ├── exporters.py │ ├── metrics.py │ └── store.py └── tests/ ├── test_data_quality.py └── test_metrics.py

📖 Documentation Preview README excerpt

ML Monitoring Suite

Production model monitoring with zero third-party dependencies. A small,

batteries-included Python package that keeps a served model healthy: prediction /

latency / throughput metrics, data-quality expectations, online drift detection

(PSI / KS / Jensen-Shannon), a stateful alert engine, Prometheus + JSON exporters,

and a ready-to-import Grafana dashboard.

Everything is pure Python standard library (3.9+), so it imports and runs in any

environment your model already runs in — no pip install required to use it.

Table of Contents

  • [Why this exists](#why-this-exists)
  • [What's included](#whats-included)
  • [Requirements](#requirements)
  • [Installation](#installation)
  • [Project structure](#project-structure)
  • [Quick start](#quick-start)
  • [The modules](#the-modules)
  • [Metrics reference](#metrics-reference)
  • [Data-quality checks](#data-quality-checks)
  • [Drift detection](#drift-detection)
  • [Alerting](#alerting)
  • [Grafana dashboard](#grafana-dashboard)
  • [Running the examples](#running-the-examples)
  • [Running the tests](#running-the-tests)
  • [FAQ](#faq)
  • [Support](#support)
  • [License](#license)

Why this exists

A model in production needs four signals: volume/throughput, latency,

errors, and drift — plus data quality underneath them all. Most teams

bolt this together from prometheus_client, great_expectations, scipy.stats,

and a pile of glue code, then fight the dependency tree every time their serving

image is rebuilt.

This suite gives you all five signals from a single stream of

record_prediction() calls, implemented from scratch on the standard library so it

can drop into a constrained serving environment (a locked-down base image, a

lambda, an air-gapped box) without adding a single dependency.

What's included

  • Prediction metrics (metrics.py) — a thread-safe PredictionMonitor that

derives volume, error rate, latency percentiles (p50/p90/p95/p99), throughput,

output-value distribution, and per-class counts from one call per request.

  • Rolling storage (store.py) — time- and size-bounded RollingWindow /

MetricStore primitives with deterministic, event-time eviction (fully testable

by passing explicit timestamps).

  • Data-quality expectations (data_quality.py) — a pandas-free, 8-expectation

framework (DataQualitySuite) for schema, null, range, type, set-membership,

cardinality and table-shape checks, with error/warning severities.

  • Drift detection (drift_monitor.py) — online Population Stability Index,

two-sample Kolmogorov-Smirnov (with an asymptotic p-value), and Jensen-Shannon

divergence, wrapped in numeric/categorical detectors and a DriftMonitor.

  • Alerting (alerting.py) — a stateful AlertManager with ThresholdRule

(with Prometheus-style for_seconds dwell) and a self-tuning z-score

... continues with setup instructions, usage examples, and more.

📄 Code Sample .py preview

src/ml_monitoring/store.py""" Rolling metric storage and time-window buffers for ML monitoring. This module provides the storage primitives the rest of the suite is built on: * :class:`RollingWindow` -- a bounded, time-aware buffer of numeric observations that exposes streaming summary statistics (count, mean, standard deviation, percentiles, rate) while never holding more than ``max_samples`` points. * :class:`MetricStore` -- a registry of named, optionally labelled rolling windows so a single serving process can track many series (latency per model, predicted value per feature) through one object. Everything here is pure Python standard library, which means it runs anywhere a model serves and is trivially unit-testable with deterministic, caller-supplied timestamps (pass ``timestamp=`` instead of relying on the wall clock). Eviction uses *event time*: the "now" of a window is the largest timestamp it has observed. A sample is dropped once it is older than ``now - window_seconds``. This keeps behaviour reproducible in tests -- feeding the same timestamps always produces the same window contents, regardless of how long the test took to run. """ from __future__ import annotations import math import time from collections import deque from dataclasses import dataclass from typing import Optional __all__ = [ "DEFAULT_WINDOW_SECONDS", "DEFAULT_MAX_SAMPLES", "WindowStats", "RollingWindow", "SeriesKey", "MetricStore", "percentile", ]
Buy Now — $39 Back to Products