Why models drift

Production data changes:

• user behavior shifts
• product changes
• seasonality
• sensor calibration changes

As a result, model performance can degrade.

Types of drift

1) Data drift (covariate shift)

Input feature distributions change.

Example:

• average income changes
• new cities appear

2) Concept drift

The relationship between X and y changes.

Example:

• fraud patterns adapt

3) Performance drift

Metrics degrade over time.

Monitoring loop

  flowchart TD
  A[Collect inputs + predictions] --> B[Track feature distributions]
  B --> C[Track model outputs]
  C --> D[Collect true outcomes (labels)]
  D --> E[Compute metrics]
  E --> F{Drift detected?}
  F -->|yes| G[Retrain / rollback]
  F -->|no| A

What to monitor in practice

• feature distribution stats (mean/std, PSI)
• prediction distribution (sudden shifts)
• business metrics (conversion, fraud loss)
• model metrics when labels arrive (AUC, F1, RMSE)

Mini-checkpoint

Why is monitoring harder than training?

• labels often arrive late (or never), so feedback loops are delayed.