Shipping a model isn’t the finish line. The world your model was trained on keeps changing after deployment — new users, new products, new seasons — and a model that scored well last quarter can quietly get worse without anyone touching its code. “Monitoring model drift” means watching the data going in and the predictions coming out so you notice that decay before it costs you, instead of finding out from an angry customer.

Why models drift

Production data changes:

• user behavior shifts
• product changes
• seasonality
• sensor calibration changes

As a result, model performance can degrade.

Types of drift

1) Data drift (covariate shift)

Input feature distributions change.

Example:

• average income changes
• new cities appear

2) Concept drift

The relationship between X and y changes.

Example:

• fraud patterns adapt

3) Performance drift

Metrics degrade over time.

Monitoring loop

  flowchart TD
  A[Collect inputs + predictions] --> B[Track feature distributions]
  B --> C[Track model outputs]
  C --> D["Collect true outcomes (labels)"]
  D --> E[Compute metrics]
  E --> F{Drift detected?}
  F -->|yes| G[Retrain / rollback]
  F -->|no| A

What to monitor in practice

• feature distribution stats (mean/std, PSI)
• prediction distribution (sudden shifts)
• business metrics (conversion, fraud loss)
• model metrics when labels arrive (AUC, F1, RMSE)

A simple drift check in code

You don’t need a fancy monitoring platform to get started — comparing basic summary statistics between the data your model trained on and the data it’s seeing now already catches a lot of real problems:

import numpy as np
 
# feature values seen during training
train_income = np.array([32000, 41000, 38000, 52000, 47000, 60000, 55000])
 
# feature values seen in production this week
live_income = np.array([61000, 72000, 68000, 80000, 75000, 90000, 85000])
 
train_mean, train_std = train_income.mean(), train_income.std()
live_mean, live_std = live_income.mean(), live_income.std()
 
# how many training standard-deviations has the live mean shifted?
shift = abs(live_mean - train_mean) / train_std
 
print(f"train mean: {train_mean:.0f}, live mean: {live_mean:.0f}")
print(f"shift: {shift:.2f} std devs")
 
if shift > 1.0:
    print("ALERT: income distribution has drifted — consider retraining")
else:
    print("OK: no significant drift detected")

import numpy as np
 
# feature values seen during training
train_income = np.array([32000, 41000, 38000, 52000, 47000, 60000, 55000])
 
# feature values seen in production this week
live_income = np.array([61000, 72000, 68000, 80000, 75000, 90000, 85000])
 
train_mean, train_std = train_income.mean(), train_income.std()
live_mean, live_std = live_income.mean(), live_income.std()
 
# how many training standard-deviations has the live mean shifted?
shift = abs(live_mean - train_mean) / train_std
 
print(f"train mean: {train_mean:.0f}, live mean: {live_mean:.0f}")
print(f"shift: {shift:.2f} std devs")
 
if shift > 1.0:
    print("ALERT: income distribution has drifted — consider retraining")
else:
    print("OK: no significant drift detected")

Running this prints something like:

train mean: 46429, live mean: 75857
shift: 3.36 std devs
ALERT: income distribution has drifted — consider retraining

train mean: 46429, live mean: 75857
shift: 3.36 std devs
ALERT: income distribution has drifted — consider retraining

In a real system you’d compute this per feature, on a schedule, and feed the result into an alert (Slack, email, a dashboard) instead of a print()print().

Visualize it

Data drift often shows up as a distribution that slowly slides away from the shape the model was trained on. Watch how the “live” distribution (orange) drifts away from the training distribution (blue) over time — the growing gap between the two peaks is exactly what a monitoring job like the one above is trying to catch:

Mini-checkpoint

Why is monitoring harder than training?

• labels often arrive late (or never), so feedback loops are delayed.

🧪 Try It Yourself

Exercise 1 – Compare Two Feature Means

Exercise 2 – Flag Drift with a Threshold

Exercise 3 – Standardize the Shift by Training Std Dev

Next

That wraps up Phase 8. You’ve taken a trained model from a saved artifact, through an API and a demo UI, into a container, and finally under continuous monitoring — the full path a model takes from a notebook to production.