Drift Detection Guide

Drift is the divergence between the data a model was trained on and the data it

now scores. This guide explains the three statistical tests this framework uses

— PSI, the Kolmogorov-Smirnov test, and the chi-square test — how to

read their output, and how to tune the thresholds that turn a raw statistic into

a PASS / WARN / FAIL verdict.

Two kinds of drift

• Data drift (covariate shift) — the distribution of the *inputs* P(X)

changes. The relationship between inputs and target may still hold, but the

model is now extrapolating into regions it saw little of during training.

• Concept drift — the relationship P(Y|X) itself changes. The same inputs

now imply a different outcome (e.g. spending €500/month meant "low risk" in

2019 and "high risk" after an economic shock).

This framework detects data drift directly, by comparing input

distributions. That is its superpower: it needs no ground-truth labels, so

it warns you *today*, weeks before the labels that would reveal concept drift

arrive. Persistent, large data drift is also the most common practical *cause*

of concept drift, so a drifting feature is a strong leading indicator even when

you cannot measure P(Y|X) directly.

Reference vs current

Every drift check compares two datasets:

• reference — the baseline you trust. Usually the training set, or the most

recent window you validated and were happy with.

• current — the window you want to test, e.g. the last 7 days of production

traffic.

from model_validation import DataDriftDetector, DriftThresholds

detector = DataDriftDetector(
    thresholds=DriftThresholds(psi_warn=0.10, psi_fail=0.25, ks_p_value=0.05),
    categorical_features=["region", "plan"],   # the rest are auto-detected
)
report = detector.detect(reference_columns, current_columns)

Input can be a dict[str, sequence] (column name → values) or a pandas

DataFrame. Numeric columns are auto-detected by dtype; name any categorical

columns explicitly via categorical_features to be safe.

Population Stability Index (PSI)

PSI is the workhorse drift metric for tabular data. It buckets a feature using

the reference quantiles, then measures how much probability mass moved

between the reference and current samples:

PSI = Σ_i (c_i − r_i) · ln(c_i / r_i)

where r_i and c_i are the proportion of the reference and current samples

that fall in bucket *i*. Because the buckets are defined by reference quantiles,

an unshifted feature scores ≈ 0. The widely used interpretation bands:

PSI is an effect-size measure: it tells you *how much* the distribution

moved, not whether the move is statistically "significant". That is exactly what

you want for a stability gate — a change large enough to matter shows up

regardless of sample size.

from model_validation import population_stability_index
psi = population_stability_index(reference["age"], current["age"], bins=10)

Notes on the implementation:

• Bins come from reference quantiles, de-duplicated so constant or

low-cardinality features never create zero-width buckets.

• The outermost buckets are open-ended, so values that fall outside the

reference range (a classic drift symptom) are captured rather than dropped.

• A small epsilon floors each proportion so PSI never divides by — or takes the

log of — zero.

Kolmogorov-Smirnov test (numeric significance)

PSI tells you the size of a move; the two-sample KS test tells you whether a

move is real. It computes D, the maximum absolute gap between the two empirical

cumulative distribution functions, and converts it to a p-value via the

Kolmogorov distribution:

from model_validation import ks_2samp
d_statistic, p_value = ks_2samp(reference["income"], current["income"])

• D ranges from 0 (identical) to 1 (disjoint). Larger means more different.
• p-value is the probability of seeing a gap that large if both samples came

from the same distribution. p < ks_p_value (default 0.05) rejects "same

distribution".

The p-value uses the standard asymptotic Kolmogorov series with the

small-sample correction term, the same approximation SciPy uses — but

implemented here in pure NumPy, so the framework has no SciPy dependency.

Why PSI *and* KS together

With large production samples, the KS test becomes hypersensitive: a difference

far too small to affect the model becomes "statistically significant" simply

because you have a million rows. This is the classic *large-n* trap. The

detector therefore flags a numeric feature only when:

PSI ≥ psi_fail            (a large move on its own), OR
(KS significant AND PSI ≥ psi_warn)   (a real move that is also big enough)

Significance answers "is it real?"; PSI answers "is it big enough to care?". You

need both to avoid a wall of false alarms.

Chi-square test (categorical significance)

For categorical features, the detector builds a 2 × k contingency table

(reference vs current counts across the union of categories) and runs a

chi-square test of homogeneity:

from model_validation import chi_square_test
statistic, p_value, dof = chi_square_test(reference["plan"], current["plan"])

The p-value is computed from the chi-square survival function via the

regularised upper incomplete gamma function — again, pure NumPy. As with numeric

features, the detector pairs significance with a categorical PSI effect-size

gate, so a new rare category or a tiny proportion wobble does not trip a false

alarm on a large sample.

Reading a drift report

detector.detect(...) returns a DriftReport. The most useful members:

report.n_features        # how many features were compared
report.n_drifted         # how many drifted
report.drift_share       # fraction drifted (0.0–1.0)
report.has_major_drift   # any feature in the "major" severity band?
report.drifted_features  # list[FeatureDriftResult]
report.get("income")     # the result for one feature
report.to_dict()         # JSON-serialisable summary

Each FeatureDriftResult carries the feature name, kind (numeric /

categorical), drifted flag, severity (none / moderate / major),

the psi, the test statistic, and the p_value. Example console output from

examples/drift_check.py:

feature         kind         severity       psi     p_value
monthly_spend   numeric      major        0.839  2.7e-295  <== DRIFT
plan            categorical  moderate     0.235  2.6e-123  <== DRIFT
age             numeric      none         0.003     0.888

Here monthly_spend shifted enough to be a hard failure, plan's category mix

moved into the warning band, and age is stable.

Tuning thresholds

DriftThresholds controls both per-feature sensitivity and how the whole check

is graded:

The DataDriftCheck grade combines per-feature severity with the overall share:

it fails if any feature shows *major* drift or the drifted share exceeds

max_drifted_share_fail; it warns if the drifted share exceeds

max_drifted_share_warn (or any feature drifted at all); otherwise it passes.

Practical tuning advice

• Start loose, tighten later. Begin with the defaults and everything as

WARN. Watch a few weeks of real traffic to learn your baseline noise floor

before you let drift block a deployment.

• Segment your reference window. Comparing against a single static training

snapshot will eventually flag seasonal-but-benign change. A rolling

"last validated window" reference adapts to legitimate evolution.

• Mind the sample size. Very small current windows make PSI and the tests

noisy; aggregate to at least a few hundred rows per feature before trusting a

verdict.

• Watch the most important features harder. Set tighter thresholds (or a

stricter gate) for the handful of features that drive most of the model's

predictions, and looser ones for long-tail features.

A standalone drift-monitoring job

from model_validation import DataDriftDetector
import sys

detector = DataDriftDetector(categorical_features=["region", "plan"])
report = detector.detect(reference, last_24h)
print(report.to_dict())
sys.exit(1 if report.has_major_drift else 0)   # alert / page on major drift

Schedule this against a rolling production window and alert on a non-zero exit.

For the full version with formatted output, see examples/drift_check.py.

Limitations to keep in mind

• Drift detection sees the *inputs*, not the *outcome*. It cannot prove the

model got worse, only that the world it operates in has changed. Pair it with

PerformanceCheck once labels arrive.

• Univariate tests (one feature at a time) miss multivariate drift, where

each feature's marginal looks fine but their *joint* distribution shifts.

Treat a clean per-feature report as necessary, not sufficient.

• A drifting feature is a signal to *investigate*, not an automatic verdict that

the model is broken — sometimes the world genuinely changed and the model

still generalises. The gate exists to make sure a human looks.

Model Validation Guide

A practical guide to validating machine-learning models before you promote them

to production and while they run there. It explains *what* to check, *why* each

check matters, and *how* to wire the checks in this framework into a gate that

can block a bad model from shipping.

Why "it scored 0.92 on the test set" is not enough

A single held-out metric tells you how a model performed on one frozen slice of

history. It says nothing about:

• whether the live data still looks like the data you trained on (drift),
• whether the model respects the logic of the problem (behavior),
• whether quality is *holding* over time (monitoring), or
• whether a regression slipped in during the last retrain (baselining).

Production model failures are rarely a sudden crash. They are a slow erosion: a

feature pipeline changes units, a customer segment grows, an upstream default

value flips from 0 to null. Aggregate accuracy can stay flat for weeks while

the model quietly stops working for the segment that matters. Validation is the

discipline of catching those failures with checks that run automatically.

The four validation layers

This framework organises checks into four layers. Each maps to a module and a

concrete Check you can add to a ValidationSuite.

You rarely need all four on day one. A pragmatic adoption order is: baseline

performance first, then drift, then behavioral tests for the cases you most fear

getting wrong.

Layer 1 — Performance vs a baseline

The most common validation mistake is comparing a model to an absolute number

("accuracy must be > 0.9") when what you really care about is *regression*

relative to the model you already trust.

PerformanceTracker compares a current metric snapshot to a baseline

snapshot and flags two kinds of problem:

• Relative degradation — the metric moved in the wrong direction by more

than max_relative_drop (for higher-is-better metrics) or

max_relative_increase (for error metrics like RMSE).

• Absolute floor violation — the metric fell below a hard floor you set,

e.g. min_accuracy=0.85, regardless of the baseline.

from model_validation import PerformanceCheck, PerformanceThresholds

check = PerformanceCheck(
    baseline={"accuracy": 0.91, "roc_auc": 0.95},
    current={"accuracy": 0.86, "roc_auc": 0.94},
    thresholds=PerformanceThresholds(min_accuracy=0.85, max_relative_drop=0.05),
)
result = check.run()  # WARN: accuracy fell 5.5% relative, but is above the floor

Record the baseline at the moment you last *validated and trusted* a model — not

the best score it ever achieved. Store it next to the model artifact and version

it. The framework's metric functions (accuracy, precision, recall, f1,

roc_auc, rmse, mae, r2) are pure NumPy, so you can compute both

snapshots the same way in training and in production with no extra dependency.

Choosing the right metric

• Imbalanced classification: do not gate on accuracy alone — a 99%-negative

problem scores 0.99 by predicting "no" forever. Gate on roc_auc, f1, or

recall for the positive class.

• Ranking / scoring: roc_auc captures ordering quality independent of the

threshold you eventually pick.

• Regression: pair rmse (penalises large errors) with mae (robust,

interpretable) and r2 (explained variance).

Layer 2 — Data drift

Drift detection is covered in depth in drift-detection.md. In short: a

DataDriftCheck compares the feature distributions of a reference dataset

(training or last-validated window) against the current production window. It is

the only layer that works without ground-truth labels, which makes it your

earliest warning system — you usually learn the true outcome (did the customer

churn? did the loan default?) weeks after you scored it, but you can measure

input drift today.

from model_validation import DataDriftCheck, DriftThresholds

check = DataDriftCheck(
    reference=train_columns,       # {"age": [...], "region": [...], ...}
    current=production_columns,
    thresholds=DriftThresholds(psi_fail=0.25, max_drifted_share_fail=0.30),
)

Layer 3 — Behavioral tests

Behavioral tests are unit tests for model logic, inspired by Ribeiro et al.'s

*Beyond Accuracy: Behavioral Testing of NLP Models* (2020). They check that the

model obeys rules a domain expert would insist on, independent of any aggregate

score. This framework ships three model-agnostic families:

1. Invariance — a change that *shouldn't* matter doesn't change the

prediction. Example: scaling an irrelevant feature, or reformatting an ID.

2. Directional expectation — a change with a *known* monotonic effect moves

the prediction the right way. Example: increasing declared income should not

*lower* a loan-approval score.

3. Minimum functionality (MFT) — a curated set of easy cases the model must

get right, like obvious-positive and obvious-negative examples.

from model_validation import (
    BehavioralCheck, InvarianceTest, DirectionalExpectationTest,
    MinimumFunctionalityTest,
)

check = BehavioralCheck([
    InvarianceTest("ignore_request_id", model.predict_proba, drop_id, samples,
                   tolerance=1e-6, min_pass_rate=1.0),
    DirectionalExpectationTest("income_helps", model.predict_proba, raise_income,
                               samples, direction=1, min_pass_rate=0.95),
    MinimumFunctionalityTest("obvious", model.predict, easy_inputs, easy_labels,
                             min_pass_rate=1.0),
])

Behavioral tests catch failures that aggregate metrics hide. A model can score

0.92 AUC and still drop its score when income rises for 8% of applicants — a bug

that is invisible in a confusion matrix but obvious (and embarrassing) in

production.

Layer 4 — The validation gate

A ValidationSuite runs every check and collapses the results into one decision

you can wire into CI/CD. Each check returns a Status of PASS, WARN, or

FAIL; the gate fails when any check reaches fail_on (default FAIL).

from model_validation import ValidationSuite, Status, to_markdown

suite = ValidationSuite("pre-promotion-gate", fail_on=Status.FAIL)
suite.add(performance_check).add(drift_check).add(behavioral_check)
result = suite.run()

print(to_markdown(result))     # human-readable report for the PR / CI log
raise SystemExit(0 if result.passed else 1)   # block promotion on failure

Use WARN for signals you want surfaced but not blocked on (mild drift, a small

metric wobble) and FAIL for release blockers (a floor violation, major drift,

a failed MFT). If you need a stricter posture, build the suite with

fail_on=Status.WARN so warnings also block.

Thresholds are policy — version them

Every threshold in this framework lives in a typed dataclass

(DriftThresholds, PerformanceThresholds, BehavioralThresholds, aggregated

by ValidationConfig). That is deliberate: thresholds are a *policy decision*,

not a constant buried in code. Serialise them to JSON, commit them next to the

model, and review changes to them like any other code change.

from model_validation import ValidationConfig

config = ValidationConfig.from_dict({
    "drift": {"psi_fail": 0.2},
    "performance": {"min_accuracy": 0.85, "max_relative_drop": 0.03},
})
config.save("validation_config.json")   # audit trail

Start conservative (loose thresholds, everything as WARN) so the gate does not

cry wolf, then tighten as you learn what "normal" looks like for your data.

A reference CI/CD wiring

retrain model
   │
   ├── compute current metrics  ─► PerformanceCheck
   ├── snapshot prod features   ─► DataDriftCheck
   └── run behavioral suite     ─► BehavioralCheck
                                       │
                              ValidationSuite.run()
                                       │
                        gate PASS ─────┴───── gate FAIL
                          │                      │
                  promote model           block + post report

The worked end-to-end version of this lives in examples/validate_model.py,

which trains a real (NumPy) classifier and runs all three checks against a

shifted production window.

Common pitfalls

• Gating on accuracy for imbalanced data. Use AUC / F1 / per-class recall.
• Comparing to the best-ever score instead of the last *trusted* baseline.
• Treating every statistically significant drift as actionable. With large

samples, trivial differences become "significant"; this framework pairs every

significance test with an effect-size (PSI) gate to avoid false alarms.

• No behavioral tests. Aggregate metrics cannot see logic bugs; add a

handful of MFTs for the cases you most fear getting wrong.

• Unversioned thresholds. If a threshold change is not in code review, your

gate's behavior is silently mutable.

Where to go next

• drift-detection.md — the statistics behind PSI, KS, and chi-square, and how

to read a drift report.

• examples/validate_model.py — full suite on a trained model.
• examples/drift_check.py — standalone drift report you can adapt as a CI job.