A structured, reusable framework for testing machine-learning models and data pipelines. Covers the full testing lifecycle: data validation, performance gates, fairness auditing, behavioral tests, and
Browse the actual product documentation and code examples included in this toolkit.
Key features of ML Testing Framework
• Data Validators — Schema checks (column presence, dtype, nullable), range validation, PSI-based distribution drift detection, completeness and duplicate checks • Performance Gates — Configurable threshold assertions for classification (accuracy, precision, recall, F1, AUC) and regression (MAE, RMSE, R², MAPE) metrics with warn-delta early warning • Fairness Auditing — Demographic parity ratio, equalized odds (TPR/FPR gap), and disparate impact ratio with stdlib-only math for portable execution • Regression Detection — Paired t-test and bootstrap confidence intervals to statistically determine if a new model is worse than the current one, with per-slice subgroup analysis • Pytest Integration — Synthetic data generators, model stubs, and temporary artifact helpers for clean test setup
Data Validators — Schema checks (column presence, dtype, nullable), range validation, PSI-based distribution drift detection, completeness and duplicate checks
Performance Gates — Configurable threshold assertions for classification (accuracy, precision, recall, F1, AUC) and regression (MAE, RMSE, R², MAPE) metrics with warn-delta early warning
Fairness Auditing — Demographic parity ratio, equalized odds (TPR/FPR gap), and disparate impact ratio with stdlib-only math for portable execution
Regression Detection — Paired t-test and bootstrap confidence intervals to statistically determine if a new model is worse than the current one, with per-slice subgroup analysis
Pytest Integration — Synthetic data generators, model stubs, and temporary artifact helpers for clean test setup
Configure ML Testing Framework parameters to see how the product works.
pip install -r requirements.txt pytest tests/ -v