← Back to all products

Model Validation Framework

$39

Model testing, data drift detection, performance monitoring, and validation gates for CI/CD.

📁 18 files🏷 v1.0.0
JSONMarkdownPythonCI/CD

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 18 files

model-validation-framework/ ├── LICENSE ├── README.md ├── examples/ │ ├── drift_check.py │ └── validate_model.py ├── guides/ │ ├── drift-detection.md │ └── model-validation-guide.md ├── src/ │ └── model_validation/ │ ├── __init__.py │ ├── behavioral.py │ ├── drift.py │ ├── performance.py │ ├── report.py │ ├── statistical_tests.py │ ├── thresholds.py │ └── validation.py └── tests/ ├── test_drift.py ├── test_performance.py └── test_validation.py

📖 Documentation Preview README excerpt

Model Validation Framework

A dependency-light framework for validating machine-learning models before

you promote them to production and while they run there. It bundles data &

concept drift detection, baseline performance monitoring, behavioral testing,

and a configurable pass/fail validation gate you can wire straight into CI/CD.

Everything is implemented in pure NumPy — the SciPy-grade statistics

(Kolmogorov-Smirnov p-values, chi-square survival function, Population Stability

Index, rank-based ROC AUC) are written from scratch, so the framework runs

anywhere NumPy is installed and has zero heavy dependencies. If you already use

scikit-learn or SciPy, the framework plugs into them cleanly (feed your own

metric dicts or model callables into the same checks).

Features

  • Data drift detection — per-feature PSI + Kolmogorov-Smirnov (numeric) and

chi-square + categorical PSI (categorical), combining effect size and

significance to avoid large-sample false alarms. Returns a structured

DriftReport.

  • Performance monitoring — compare a current metric snapshot to a trusted

baseline; detect relative degradation and absolute-floor violations.

  • Behavioral tests — model-agnostic invariance, directional-expectation, and

minimum-functionality tests (CheckList-style) that catch logic bugs aggregate

metrics hide.

  • Validation gate — a ValidationSuite aggregates any mix of checks into a

single PASS / WARN / FAIL decision, with a configurable fail threshold.

  • Markdown & JSON reports — drop a readable report into a PR comment or a CI

log; emit JSON for dashboards and audit trails.

  • Typed, versionable thresholds — all policy lives in dataclasses

(ValidationConfig) you can serialise to JSON and commit next to the model.

  • NumPy metric functions — accuracy, precision, recall, f1,

roc_auc, rmse, mae, r2 with no scikit-learn requirement.

Requirements

  • Python 3.9+
  • NumPy (the only hard dependency)
  • Optional: pandas (drift detection accepts DataFrames as well as dict-of-

columns), scikit-learn / SciPy (interop — pass their metrics/models in)


pip install numpy        # required
pip install pandas       # optional, for DataFrame inputs

Quick Start

The package lives under src/. Run the examples straight from the product root

(they add src/ to the path for you):


# 1. Standalone drift report on synthetic reference vs production data
python examples/drift_check.py

# 2. Full validation suite on a trained (NumPy) classifier
python examples/validate_model.py

To use it in your own project, put src/ on your PYTHONPATH (or copy the

... continues with setup instructions, usage examples, and more.

📄 Code Sample .py preview

src/model_validation/statistical_tests.py"""Self-contained statistical primitives for model validation. This module implements the statistical machinery the framework needs *without* depending on SciPy or scikit-learn, so the validation suite runs anywhere NumPy is installed. Every function here is exercised by the test-suite against known closed-form values. What lives here and why it matters for drift / monitoring: * :func:`population_stability_index` — PSI, the workhorse metric for tabular data drift. Buckets a feature using the *reference* distribution and measures how much probability mass moved. * :func:`ks_2samp` — two-sample Kolmogorov-Smirnov test for continuous features. Returns the D statistic and an asymptotic p-value computed from the Kolmogorov distribution (same approximation SciPy uses). * :func:`chi_square_test` — two-sample chi-square test of homogeneity for categorical features, with the p-value derived from the regularised upper incomplete gamma function. * :func:`roc_auc_score` — rank-based (Mann-Whitney U) ROC AUC that correctly averages ties; used by the performance module when SciPy/sklearn are absent. The incomplete-gamma and Kolmogorov implementations follow the standard Numerical-Recipes series / continued-fraction expansions. """ from __future__ import annotations import math from typing import Sequence import numpy as np # A p-value below this is the conventional "statistically significant" cut. DEFAULT_ALPHA: float = 0.05 # Number of histogram buckets used by PSI unless the caller overrides it. DEFAULT_PSI_BINS: int = 10 # Floor applied to bucket proportions so PSI never divides by / logs zero. PSI_EPSILON: float = 1e-6
Buy Now — $39 Back to Products