← Back to all products

Model Validation Framework

$39

Model testing, data drift detection, performance monitoring, and validation gates for CI/CD.

📁 18 files🏷 v1.0.0
JSONMarkdownPythonCI/CD

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 18 files

model-validation-framework/ ├── LICENSE ├── README.md ├── examples/ │ ├── drift_check.py │ └── validate_model.py ├── guides/ │ ├── drift-detection.md │ └── model-validation-guide.md ├── src/ │ └── model_validation/ │ ├── __init__.py │ ├── behavioral.py │ ├── drift.py │ ├── performance.py │ ├── report.py │ ├── statistical_tests.py │ ├── thresholds.py │ └── validation.py └── tests/ ├── test_drift.py ├── test_performance.py └── test_validation.py

📖 Documentation Preview README excerpt

Model Validation Framework

A dependency-light framework for validating machine-learning models before

you promote them to production and while they run there. It bundles data &

concept drift detection, baseline performance monitoring, behavioral testing,

and a configurable pass/fail validation gate you can wire straight into CI/CD.

Everything is implemented in pure NumPy — the SciPy-grade statistics

(Kolmogorov-Smirnov p-values, chi-square survival function, Population Stability

Index, rank-based ROC AUC) are written from scratch, so the framework runs

anywhere NumPy is installed and has zero heavy dependencies. If you already use

scikit-learn or SciPy, the framework plugs into them cleanly (feed your own

metric dicts or model callables into the same checks).

Features

  • Data drift detection — per-feature PSI + Kolmogorov-Smirnov (numeric) and

chi-square + categorical PSI (categorical), combining effect size and

significance to avoid large-sample false alarms. Returns a structured

DriftReport.

  • Performance monitoring — compare a current metric snapshot to a trusted

baseline; detect relative degradation and absolute-floor violations.

  • Behavioral tests — model-agnostic invariance, directional-expectation, and

minimum-functionality tests (CheckList-style) that catch logic bugs aggregate

metrics hide.

  • Validation gate — a ValidationSuite aggregates any mix of checks into a

single PASS / WARN / FAIL decision, with a configurable fail threshold.

  • Markdown & JSON reports — drop a readable report into a PR comment or a CI

log; emit JSON for dashboards and audit trails.

  • Typed, versionable thresholds — all policy lives in dataclasses

(ValidationConfig) you can serialise to JSON and commit next to the model.

  • NumPy metric functions — accuracy, precision, recall, f1,

roc_auc, rmse, mae, r2 with no scikit-learn requirement.

Requirements

  • Python 3.9+
  • NumPy (the only hard dependency)
  • Optional: pandas (drift detection accepts DataFrames as well as dict-of-

columns), scikit-learn / SciPy (interop — pass their metrics/models in)


pip install numpy        # required
pip install pandas       # optional, for DataFrame inputs

Quick Start

The package lives under src/. Run the examples straight from the product root

(they add src/ to the path for you):


# 1. Standalone drift report on synthetic reference vs production data
python examples/drift_check.py

# 2. Full validation suite on a trained (NumPy) classifier
python examples/validate_model.py

To use it in your own project, put src/ on your PYTHONPATH (or copy the

... continues with setup instructions, usage examples, and more.

📄 Code Sample .py preview

src/model_validation/behavioral.py """Behavioral testing for ML models (CheckList-style). Aggregate metrics like accuracy tell you *how often* a model is right but not *why* it fails or whether it respects the logic of the problem. Behavioral tests (popularised by Ribeiro et al., "Beyond Accuracy: Behavioral Testing of NLP Models", 2020) probe specific, human-understandable capabilities. This module implements three model-agnostic families that apply to any predictor exposed as a Python callable: * :class:`InvarianceTest` — applying a label-preserving perturbation must *not* change the prediction (e.g. scaling an irrelevant feature, reordering inputs). * :class:`DirectionalExpectationTest` — a perturbation with a known monotonic effect must move the prediction in the expected direction (e.g. increasing income should not *decrease* a loan-approval score). * :class:`MinimumFunctionalityTest` — a curated set of simple cases the model must get right, like unit tests for model behavior. Each test consumes a ``predict_fn`` returning a NumPy array (labels for invariance/MFT, continuous scores for directional tests) and reports a pass-rate plus the indices of the failing cases for debugging. """ from __future__ import annotations from dataclasses import dataclass, field from typing import Callable, Sequence import numpy as np PredictFn = Callable[[np.ndarray], np.ndarray] PerturbFn = Callable[[np.ndarray], np.ndarray] @dataclass class BehavioralTestResult: """Outcome of a single behavioral test.""" name: str test_type: str n_cases: int n_passed: int min_pass_rate: float failing_indices: list[int] = field(default_factory=list) @property def pass_rate(self) -> float: return self.n_passed / self.n_cases if self.n_cases else 1.0 @property def passed(self) -> bool: # ... 148 more lines ...
Buy Now — $39 Back to Products