← Back to all products

Model Monitoring Dashboard

$39

Drift detection, performance monitoring, and alerting for deployed models with Grafana dashboards and alerting rules.

📁 36 files

JSONMarkdownYAMLPythonGrafanaPrometheus

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 36 files

model-monitoring-dashboard/ ├── LICENSE ├── README.md ├── configs/ │ ├── alert_rules.yaml │ ├── monitoring_config.yaml │ └── prometheus.yaml ├── dashboards/ │ ├── drift_analysis.json │ └── model_overview.json ├── free-sample.zip ├── guide/ │ ├── 01-drift_detection_explained.md │ └── 02-setup_guide.md ├── guides/ │ ├── drift_detection_explained.md │ └── setup_guide.md ├── index.html ├── requirements.txt ├── src/ │ ├── __init__.py │ ├── __pycache__/ │ │ ├── __init__.cpython-312.pyc │ │ ├── alert_engine.cpython-312.pyc │ │ ├── concept_drift.cpython-312.pyc │ │ ├── drift_detectors.cpython-312.pyc │ │ ├── metric_exporters.cpython-312.pyc │ │ ├── performance_monitor.cpython-312.pyc │ │ ├── prediction_logger.cpython-312.pyc │ │ └── stats_utils.cpython-312.pyc │ ├── alert_engine.py │ ├── concept_drift.py │ ├── drift_detectors.py │ ├── metric_exporters.py │ ├── performance_monitor.py │ ├── prediction_logger.py │ └── stats_utils.py └── tests/ ├── __pycache__/ │ ├── test_alert_engine.cpython-312.pyc │ ├── test_drift_detectors.cpython-312.pyc │ └── test_performance_monitor.cpython-312.pyc ├── test_alert_engine.py ├── test_drift_detectors.py └── test_performance_monitor.py

📖 Documentation Preview README excerpt

Model Monitoring Dashboard

Detect data drift, track model performance, and get alerted before your ML models silently degrade in production.

Models don't fail with stack traces — they fail by quietly returning worse predictions until someone notices revenue dropped. This toolkit gives you statistical drift detection, rolling performance metrics, structured prediction logging, and Grafana dashboards to catch degradation early.

What's Inside

Component	What It Does
Drift Detectors	PSI, KS-test, and chi-squared tests for numeric/categorical feature drift. Pure-Python math with stdlib fallbacks.
Concept Drift	ADWIN and Page-Hinkley online detectors for when the feature-target relationship changes (not just the inputs).
Performance Monitor	Sliding-window accuracy, precision, recall, F1, MAE, RMSE, R² with baseline comparison and degradation alerts.
Prediction Logger	Structured JSONL logging of every prediction with features, confidence, latency, and ground-truth join.
Prometheus Exporter	Expose all metrics on a `/metrics` endpoint for Prometheus scraping.
Alert Engine	Rule-based alerting with cooldowns, severity levels, consecutive-check filtering, and webhook/file/log channels.
Grafana Dashboards	Two importable JSON dashboards: model health overview and drift analysis deep-dive.
Alert Configs	YAML-defined alert rules with documented thresholds for drift, performance, and operational metrics.

Quick Start


pip install -r requirements.txt
python -m src.drift_detectors    # Run drift detection demo
python -m src.concept_drift      # Run concept drift detection demo
python -m src.performance_monitor  # Run performance monitoring demo
python -m src.alert_engine       # Run alert engine demo

Each module is executable standalone — run it to see the algorithms working on synthetic data with known drift patterns.

File Tree


model-monitoring-dashboard/
├── README.md
├── LICENSE
├── requirements.txt
├── src/
│   ├── __init__.py                    # Package description and module index
│   ├── stats_utils.py                 # Pure-Python CDF, histogram, KS math, chi-squared
│   ├── drift_detectors.py            # PSI, KS-test, chi-squared drift detectors
│   ├── concept_drift.py              # ADWIN + Page-Hinkley online concept drift
│   ├── performance_monitor.py        # Sliding-window classification & regression metrics
│   ├── prediction_logger.py          # JSONL prediction logging + ground-truth join
│   ├── metric_exporters.py           # Prometheus metric exporter + HTTP server
│   └── alert_engine.py               # Rule-based alerting with cooldown/escalation
├── dashboards/
│   ├── model_overview.json           # Grafana: accuracy, F1, latency, drift, throughput
│   └── drift_analysis.json           # Grafana: PSI heatmap, top drifted features, timeline
├── configs/
│   ├── alert_rules.yaml              # 11 pre-defined alert rules with thresholds
│   ├── monitoring_config.yaml        # Central config: models, baselines, drift settings
│   └── prometheus.yaml               # Prometheus scrape configuration
├── tests/
│   ├── test_drift_detectors.py       # Statistical correctness tests with known distributions
│   ├── test_performance_monitor.py   # Classification/regression metric accuracy
│   └── test_alert_engine.py          # Rule evaluation, cooldowns, notifications

*... continues with setup instructions, usage examples, and more.*

📄 Code Sample .py preview

src/alert_engine.py """ Alert Engine — Rule-Based Alerting for ML Monitoring ===================================================== Evaluates alert rules against monitoring metrics and fires notifications via configurable channels (webhook, email stub, log). Includes: - YAML-driven rule definitions (see configs/alert_rules.yaml) - Cooldown periods to prevent alert storms - Severity escalation (warning → critical → page) - Alert history with deduplication - Composable conditions (threshold, rate-of-change, consecutive) Alert flow: Metrics arrive → Rules evaluated → Conditions checked → Cooldown filter → Fire notification → Record in history The alerting is intentionally decoupled from the metric collection. You feed metrics to the engine explicitly; it doesn't poll or scrape. This makes it testable and keeps the architecture composable. """ from __future__ import annotations import json import logging import time import urllib.request from dataclasses import dataclass, field from datetime import datetime, timezone from pathlib import Path from typing import Any, Callable, Dict, List, Optional logger = logging.getLogger(__name__) @dataclass class AlertRule: """Definition of a single alert rule. Attributes: name: Human-readable rule name. metric_name: Key in the metrics dict to evaluate. condition: One of 'gt' (>), 'lt' (<), 'gte' (>=), 'lte' (<=), 'eq' (==). threshold: Value to compare against. severity: 'warning', 'critical', or 'page'. cooldown_seconds: Minimum time between firings of this rule. description: Explanation shown in the alert message. labels: Additional key-value pairs for routing/filtering. """ name: str # ... 391 more lines ...

Buy Now — $39 Back to Products