Drift Detection Explained

A practical guide to understanding and implementing drift detection in ML systems.

Why Models Degrade in Production

A model trained on historical data makes one critical assumption: the future will look like the past. When that assumption breaks, the model's predictions become unreliable. This is "drift."

There are three distinct types, and they require different detection strategies:

1. Data Drift (Covariate Shift)

What changes: The distribution of input features.

Example: Your churn model was trained on customers aged 25-45, but a marketing campaign brings in users aged 55+. The model has never seen this population and may produce unreliable predictions.

Detection: Compare the feature distributions between training data and production data using statistical tests like PSI or KS-test.

Key insight: Data drift can happen WITHOUT the model being wrong — the model might still make correct predictions for the new population. But it's a warning sign that the model is operating outside its training domain.

2. Concept Drift

What changes: The relationship between features and the target variable.

Example: During a recession, the factors that predict customer churn change. Previously, low engagement predicted churn. Now, even engaged customers are churning due to budget cuts. The model's learned rules are no longer valid.

Detection: Monitor prediction errors over time. If the error rate increases while the input distribution stays stable, it's likely concept drift. ADWIN and Page-Hinkley are online algorithms that detect this without storing all historical data.

Key insight: Concept drift is more dangerous than data drift because even familiar-looking inputs produce wrong outputs. The model is confidently wrong.

3. Label Drift (Prior Probability Shift)

What changes: The proportion of positive/negative cases.

Example: A fraud detection model trained when 1% of transactions were fraudulent starts seeing 5% fraud (new attack vector). The model's threshold, calibrated for 1% base rate, now produces too many false negatives.

Detection: Track the prediction distribution and the ground-truth positive rate over time.

Statistical Tests in This Toolkit

Population Stability Index (PSI)

PSI measures how much a distribution has shifted by comparing binned probability distributions between a reference (training) set and the current (production) set.

How it works:

1. Bin the reference data into N equal-width buckets (default: 10)

2. Count what proportion of reference data falls in each bin

3. Count what proportion of current data falls in each bin

4. For each bin: psi_i = (P_current - P_reference) × ln(P_current / P_reference)

5. Sum across all bins: PSI = Σ psi_i

Interpretation:

Strengths:

• Easy to interpret (single number per feature)
• Industry standard in financial services / credit scoring
• Works well for continuous features with enough data

Weaknesses:

• Sensitive to bin count choice
• Can miss multimodal shifts within bins
• Requires a fixed reference dataset

Kolmogorov-Smirnov Test (KS Test)

The KS test compares two empirical CDFs and measures the maximum vertical distance between them. It's non-parametric — no assumptions about the underlying distribution.

How it works:

1. Sort both samples

2. Build the empirical CDF for each: F(x) = (count of values ≤ x) / n

3. Find the point where the CDFs are furthest apart: D = max |F_ref(x) - F_cur(x)|

4. Compare D against a critical value that depends on sample sizes and significance level

Interpretation:

• If D > critical value → reject null hypothesis (distributions are different)
• The critical value at α=0.05 is approximately 1.36 × √((n1+n2)/(n1×n2))

Strengths:

• Distribution-free
• Sensitive to changes in location, scale, and shape
• Mathematically rigorous (p-value interpretation)

Weaknesses:

• Less interpretable than PSI ("how much" is it drifting?)
• Sensitive to sample size (large samples = everything is "significant")
• Doesn't tell you WHERE the distribution shifted

Chi-Squared Test (Categorical Features)

For categorical features (country, device type, subscription tier), we use Pearson's chi-squared test to compare category frequencies.

How it works:

1. Count occurrences of each category in reference and current data

2. Scale reference counts to match current sample size (expected counts)

3. Compute: χ² = Σ (observed_i - expected_i)² / expected_i

4. Compare against chi-squared distribution with (k-1) degrees of freedom

When to use: Any feature with a small number of discrete values. For features with 100+ categories, consider grouping rare categories or using a different test.

Concept Drift Detectors

ADWIN (Adaptive Windowing)

ADWIN maintains a variable-length window of recent observations (typically prediction errors). At each step, it checks whether any split of the window shows a statistically significant mean difference.

Key properties:

• Adapts automatically — grows when stable, shrinks when drift occurs
• Memory-efficient via exponential histogram compression
• Theoretical guarantees on false positive/negative rates
• Controlled by delta parameter (smaller = fewer false alarms)

When to use: Gradual drift where you want automatic sensitivity adjustment.

Page-Hinkley Test

A cumulative sum (CUSUM) test that detects changes in the mean of a stream. It maintains a running sum of deviations from the mean and triggers when the cumulative deviation exceeds a threshold.

Key properties:

• Simple and fast (O(1) per observation)
• Two parameters: delta (dead zone) and lambda (threshold)
• Good for detecting abrupt mean shifts
• Resets after detection to find subsequent change points

When to use: Abrupt concept drift (sudden model degradation).

Decision Tree: Which Test to Use

Is the feature categorical?
├── YES → Chi-Squared Test
└── NO (continuous) →
    ├── Do you need a single interpretability score? → PSI
    ├── Do you need rigorous statistical significance? → KS Test
    └── For online streaming detection → ADWIN or Page-Hinkley

Running Drift Analysis in Practice

How Often?

How Much Data?

Both reference and current samples should have at least 200-500 observations per feature for reliable test results. With fewer than 100, statistical tests lack power and may miss real drift.

What To Do When Drift Is Detected

1. Investigate before retraining. Drift might be a data quality issue, not a real distribution change.

2. Check upstream data sources. Schema changes, ETL bugs, and missing imputation are common root causes.

3. Compare drift magnitude across features. If one feature drifts while others are stable, it's likely a data issue rather than a population shift.

4. Monitor performance alongside drift. Data drift without performance degradation may not require action.

5. When retraining, use a blend of old and new data unless the old distribution is completely irrelevant.

Setup Guide — Model Monitoring Dashboard

Step-by-step instructions for integrating the monitoring toolkit into your ML serving infrastructure.

Prerequisites

• Python 3.10+
• A deployed ML model (any framework — sklearn, PyTorch, TensorFlow, etc.)
• Prometheus server (for metrics scraping)
• Grafana (for dashboards — 9.0+ recommended)

Quick Start (5 Minutes)

1. Install Dependencies

pip install -r requirements.txt

2. Instrument Your Prediction Code

Add monitoring hooks to your model serving endpoint:

from src.drift_detectors import MultiFeatureDriftScanner
from src.performance_monitor import ClassificationMonitor, ClassificationMetrics
from src.prediction_logger import PredictionLogger
from src.metric_exporters import MLMetricsExporter
from src.alert_engine import AlertEngine, AlertRule, LogNotificationChannel

# --- Set up components ---

# Prediction logging
pred_logger = PredictionLogger(
    log_dir="prediction_logs",
    buffer_size=100,
    model_name="churn_model",
    model_version="v2.3",
)

# Performance monitoring with baseline from validation
baseline = ClassificationMetrics(accuracy=0.92, precision=0.89, recall=0.87, f1_score=0.88)
perf_monitor = ClassificationMonitor(window_size=1000, baseline_metrics=baseline)

# Prometheus exporter
exporter = MLMetricsExporter()
exporter.start_http_server(port=9090)

# --- In your prediction handler ---

def predict(features: dict) -> dict:
    # Your model inference
    prediction = model.predict(features)
    confidence = model.predict_proba(features)

    # Log prediction
    pred_logger.log(
        features=features,
        prediction=prediction,
        confidence=confidence,
    )

    # Update performance (when ground truth arrives later)
    # perf_monitor.add_prediction(y_true=actual, y_pred=prediction)

    # Export to Prometheus
    exporter.record_prediction("churn_model", "v2.3", confidence=confidence)

    return {"prediction": prediction, "confidence": confidence}

3. Run Scheduled Drift Analysis

Create a cron job or scheduled task that runs drift analysis against reference data:

import json

from src.drift_detectors import MultiFeatureDriftScanner

# Load reference data (from training set)
with open("reference_data.json") as f:
    reference = json.load(f)

# Load recent production data (from prediction logs)
# ... your data loading logic here ...
current = load_recent_predictions(hours=24)

scanner = MultiFeatureDriftScanner(
    psi_threshold=0.25,
    ks_alpha=0.05,
)

results = scanner.scan_numeric(reference["numeric"], current["numeric"])
results += scanner.scan_categorical(reference["categorical"], current["categorical"])

report = scanner.generate_report(results)

if report["status"] == "drift_detected":
    print(f"DRIFT: {report['features_drifted']}/{report['features_tested']} features")

4. Import Grafana Dashboards

1. Open Grafana → Dashboards → Import

2. Upload dashboards/model_overview.json

3. Upload dashboards/drift_analysis.json

4. Select your Prometheus data source

5. The dashboards auto-discover models via label queries

5. Configure Alerts

Edit configs/alert_rules.yaml to match your thresholds, then load them:

import yaml
from src.alert_engine import AlertEngine, LogNotificationChannel, WebhookNotificationChannel

with open("configs/alert_rules.yaml") as f:
    config = yaml.safe_load(f)

engine = AlertEngine(consecutive_required=2)
engine.load_rules_from_dicts(config["rules"])
engine.add_channel(LogNotificationChannel())

# Optional: add Slack webhook
# engine.add_channel(WebhookNotificationChannel("https://hooks.example.com/YOUR_WEBHOOK"))

Architecture Overview

┌──────────────────┐     ┌──────────────┐     ┌──────────────┐
│  Model Server    │────▶│  Prediction  │────▶│  JSONL Logs  │
│  (your code)     │     │  Logger      │     │  (on disk)   │
└────────┬─────────┘     └──────────────┘     └──────────────┘
         │
         │ metrics
         ▼
┌──────────────────┐     ┌──────────────┐     ┌──────────────┐
│  Metric Exporter │────▶│  Prometheus  │────▶│   Grafana    │
│  (:9090/metrics) │     │  (scrapes)   │     │  dashboards  │
└──────────────────┘     └──────────────┘     └──────────────┘
                                                     │
┌──────────────────┐     ┌──────────────┐            │
│  Drift Scanner   │────▶│ Alert Engine │────────────┘
│  (scheduled)     │     │ (webhook/log)│
└──────────────────┘     └──────────────┘

File Overview

Troubleshooting

Metrics endpoint returns empty:

• Check that prometheus_client is installed: pip install prometheus-client
• Verify the port isn't blocked: curl http://localhost:9090/metrics

Drift tests show no drift on obviously shifted data:

• Increase sample size (need 200+ observations per feature)
• Lower the PSI threshold or KS alpha
• Check that reference data is from the training set, not an old production window

Alert not firing:

• Check cooldown period — the alert may be silenced
• Verify the metric name in the rule matches exactly
• Check the consecutive_required setting

Support

Questions or issues? Contact support@datanest.dev