Why Models Degrade in Production

A model trained on historical data makes one critical assumption: the future will look like the past. When that assumption breaks, the model's predictions become unreliable. This is "drift."

There are three distinct types, and they require different detection strategies:

1. Data Drift (Covariate Shift)

What changes: The distribution of input features.

Example: Your churn model was trained on customers aged 25-45, but a marketing campaign brings in users aged 55+. The model has never seen this population and may produce unreliable predictions.

Detection: Compare the feature distributions between training data and production data using statistical tests like PSI or KS-test.

Key insight: Data drift can happen WITHOUT the model being wrong — the model might still make correct predictions for the new population. But it's a warning sign that the model is operating outside its training domain.

2. Concept Drift

What changes: The relationship between features and the target variable.

Example: During a recession, the factors that predict customer churn change. Previously, low engagement predicted churn. Now, even engaged customers are churning due to budget cuts. The model's learned rules are no longer valid.

Detection: Monitor prediction errors over time. If the error rate increases while the input distribution stays stable, it's likely concept drift. ADWIN and Page-Hinkley are online algorithms that detect this without storing all historical data.

Key insight: Concept drift is more dangerous than data drift because even familiar-looking inputs produce wrong outputs. The model is confidently wrong.

3. Label Drift (Prior Probability Shift)

What changes: The proportion of positive/negative cases.

Example: A fraud detection model trained when 1% of transactions were fraudulent starts seeing 5% fraud (new attack vector). The model's threshold, calibrated for 1% base rate, now produces too many false negatives.

Detection: Track the prediction distribution and the ground-truth positive rate over time.

Included with Model Monitoring Dashboard

Ready for the complete Model Monitoring Dashboard?