← Back to all products

ML Testing Framework

$29

Test ML code: data validation, model performance, fairness checks, integration tests, and regression detection.

📁 35 files
MarkdownYAMLPython

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 35 files

ml-testing-framework/ ├── LICENSE ├── README.md ├── configs/ │ ├── thresholds.yaml │ └── validation_rules.yaml ├── free-sample.zip ├── guide/ │ ├── 01-fairness-testing.md │ └── 02-ml-testing-strategy.md ├── guides/ │ ├── fairness-testing.md │ └── ml-testing-strategy.md ├── index.html ├── requirements.txt ├── src/ │ ├── __init__.py │ ├── __pycache__/ │ │ ├── __init__.cpython-312.pyc │ │ ├── behavioral.cpython-312.pyc │ │ ├── fairness.cpython-312.pyc │ │ ├── fixtures.cpython-312.pyc │ │ ├── performance.cpython-312.pyc │ │ ├── regression.cpython-312.pyc │ │ └── validators.cpython-312.pyc │ ├── behavioral.py │ ├── fairness.py │ ├── fixtures.py │ ├── performance.py │ ├── regression.py │ └── validators.py └── tests/ ├── __pycache__/ │ ├── test_behavioral.cpython-312.pyc │ ├── test_fairness.cpython-312.pyc │ ├── test_performance.cpython-312.pyc │ ├── test_regression.cpython-312.pyc │ └── test_validators.cpython-312.pyc ├── test_behavioral.py ├── test_fairness.py ├── test_performance.py ├── test_regression.py └── test_validators.py

📖 Documentation Preview README excerpt

ML Testing Framework

A structured, reusable framework for testing machine-learning models and data pipelines. Covers the full testing lifecycle: data validation, performance gates, fairness auditing, behavioral tests, and regression detection between model versions.

Features

  • Data Validators — Schema checks (column presence, dtype, nullable), range validation, PSI-based distribution drift detection, completeness and duplicate checks
  • Performance Gates — Configurable threshold assertions for classification (accuracy, precision, recall, F1, AUC) and regression (MAE, RMSE, R², MAPE) metrics with warn-delta early warning
  • Fairness Auditing — Demographic parity ratio, equalized odds (TPR/FPR gap), and disparate impact ratio with stdlib-only math for portable execution
  • Behavioral Tests — Invariance tests (neutral perturbations shouldn't change output), directional expectation tests (increasing X should increase Y), metamorphic tests (input transforms should produce predictable output transforms), and noise robustness
  • Regression Detection — Paired t-test and bootstrap confidence intervals to statistically determine if a new model is worse than the current one, with per-slice subgroup analysis
  • Pytest Integration — Synthetic data generators, model stubs, and temporary artifact helpers for clean test setup

File Structure


ml-testing-framework/
├── README.md
├── LICENSE
├── requirements.txt
├── src/
│   ├── __init__.py            # Package init with version
│   ├── validators.py          # Schema, range, drift, completeness checks
│   ├── performance.py         # Metric gates and auto-computation
│   ├── fairness.py            # Demographic parity, equalized odds, subgroup metrics
│   ├── behavioral.py          # Invariance, directional, metamorphic tests
│   ├── regression.py          # Paired t-test, bootstrap, RegressionDetector
│   └── fixtures.py            # Synthetic data, model stubs, temp artifacts
├── tests/
│   ├── test_validators.py     # 14 tests: schema, PSI, completeness, duplicates
│   ├── test_performance.py    # 12 tests: metrics, gates, auto-compute, parsing
│   ├── test_fairness.py       # 10 tests: DP ratio, EO gap, audit, subgroups
│   ├── test_behavioral.py     # 8 tests: invariance, directional, metamorphic, noise
│   └── test_regression.py     # 9 tests: t-test, bootstrap, detector, slices
├── configs/
│   ├── thresholds.yaml        # Performance, fairness, and behavioral thresholds
│   └── validation_rules.yaml  # Per-column schema definition example
└── guides/
    ├── ml-testing-strategy.md # Layered testing pyramid with code examples
    └── fairness-testing.md    # Fairness metrics deep dive and remediation

Quick Start


pip install -r requirements.txt
pytest tests/ -v

Usage

Data Validation


from src.validators import SchemaValidator, ColumnRule, check_distribution_drift

rules = [
    ColumnRule("age", dtype="integer", nullable=False, min_value=0, max_value=120),
    ColumnRule("income", dtype="float", nullable=True, min_value=0),
]

*... continues with setup instructions, usage examples, and more.*

📄 Code Sample .py preview

src/behavioral.py """ behavioral — Invariance, directional-expectation, and metamorphic tests. Traditional unit tests assert on *specific* outputs for *specific* inputs. Behavioral tests instead assert on *properties* that should hold across families of inputs. This is much more natural for ML models where the exact output is stochastic or hard to specify. Three test families ------------------- 1. **Invariance tests** — Perturbing the input in a semantically neutral way should NOT change the prediction. Example: adding whitespace to a text classifier's input shouldn't flip the sentiment. 2. **Directional expectation tests** — Perturbing the input in a known direction should move the prediction in a predictable direction. Example: increasing a credit applicant's income should never *decrease* their predicted creditworthiness. 3. **Metamorphic tests** — A transformation of the input should produce a transformation of the output that satisfies a known relation. Example: doubling all prices in a demand-forecasting model should roughly halve predicted demand (within some tolerance). The test runner accepts any callable ``model_fn(input) -> output`` so it works with sklearn estimators, PyTorch modules, REST API wrappers, etc. """ from __future__ import annotations import copy import logging from dataclasses import dataclass, field from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple import numpy as np logger = logging.getLogger(__name__) # --------------------------------------------------------------------------- # Result types # --------------------------------------------------------------------------- @dataclass class BehavioralTestResult: """Outcome of a single behavioral test case.""" test_name: str test_type: str # "invariance" | "directional" | "metamorphic" # ... 275 more lines ...
Buy Now — $29 Back to Products