What you’ll learn

• which data corruptions sklearn catches for you, and which it serves happily: 0.8500 → 0.7125
• why passing a numpy array instead of a DataFrame disables the only check you were getting
• a schema contract in about forty lines that catches 6 of 6 corruptions in 6.65 ms
• how to set range and shift thresholds, and why they fire at +0.75 when accuracy has not moved
• the honest limit: a validation alarm is not a performance prediction
• where panderapandera, Great Expectations and pydanticpydantic fit, and what they do not solve

Six corruptions, one fitted pipeline

A StandardScalerStandardScaler plus logistic regression, fitted once on 2,800 rows and scoring 0.8500 on 1,200 clean test rows. Now hand it the same rows, corrupted six ways that all have production analogues.

figure4 corruptions raise. 3 are silent, and the worst costs 0.1375.matplotlib

Three observations that determine everything below.

sklearn’s DataFrame name check is the cheapest validation you will ever get. It caught four of seven. Keep your serving path on DataFrames with real column names for that reason alone.

df.to_numpy()df.to_numpy() throws it away. The same reordering that raised a ValueErrorValueError as a DataFrame silently costs 0.0567 as an array. Every .values.values or .to_numpy().to_numpy() between your feature code and your model is a place where column order becomes a convention rather than a contract.

Nothing in sklearn checks values. A column in the wrong units is the right shape, the right dtype and the right name. Only a comparison against what training looked like can catch it.

A schema contract in forty lines

Fit the contract on the training frame, then check every batch against it. Everything below is pandaspandas and nothing else.

def fit_schema(df, quantile=0.001):
    """Everything about the training frame a serving batch should still satisfy."""
    return {
        "columns": list(df.columns),
        "dtypes": {c: str(df[c].dtype) for c in df.columns},
        "lower": {c: float(df[c].quantile(quantile)) for c in df.columns},
        "upper": {c: float(df[c].quantile(1 - quantile)) for c in df.columns},
        "mean": {c: float(df[c].mean()) for c in df.columns},
        "std": {c: float(df[c].std()) for c in df.columns},
        "null_rate": {c: float(df[c].isna().mean()) for c in df.columns},
        "rows_fitted": len(df),
    }
 
 
def validate(df, schema, max_out_of_range=0.01, max_shift=0.5,
             max_null_increase=0.01):
    """One message per violated expectation. Empty list means the batch is usable."""
    problems = []
    if list(df.columns) != schema["columns"]:
        missing = [c for c in schema["columns"] if c not in df.columns]
        extra = [c for c in df.columns if c not in schema["columns"]]
        if missing:
            problems.append(f"missing columns: {missing}")
        if extra:
            problems.append(f"unexpected columns: {extra}")
        if not missing and not extra:
            problems.append("columns in a different ORDER")
 
    for c in schema["columns"]:
        if c not in df.columns:
            continue
        if str(df[c].dtype) != schema["dtypes"][c]:
            problems.append(f"{c}: dtype {df[c].dtype} != {schema['dtypes'][c]}")
        nulls = float(df[c].isna().mean())
        if nulls > schema["null_rate"][c] + max_null_increase:
            problems.append(f"{c}: null rate {nulls:.4f} "
                            f"(fitted {schema['null_rate'][c]:.4f})")
        col = df[c].dropna()
        if len(col) == 0:
            continue
        out = float(((col < schema["lower"][c]) | (col > schema["upper"][c])).mean())
        if out > max_out_of_range:
            problems.append(f"{c}: {out:.2%} of values outside the fitted range")
        shift = abs(col.mean() - schema["mean"][c]) / max(schema["std"][c], 1e-12)
        if shift > max_shift:
            problems.append(f"{c}: mean shifted {shift:.2f} sd")
    return problems

def fit_schema(df, quantile=0.001):
    """Everything about the training frame a serving batch should still satisfy."""
    return {
        "columns": list(df.columns),
        "dtypes": {c: str(df[c].dtype) for c in df.columns},
        "lower": {c: float(df[c].quantile(quantile)) for c in df.columns},
        "upper": {c: float(df[c].quantile(1 - quantile)) for c in df.columns},
        "mean": {c: float(df[c].mean()) for c in df.columns},
        "std": {c: float(df[c].std()) for c in df.columns},
        "null_rate": {c: float(df[c].isna().mean()) for c in df.columns},
        "rows_fitted": len(df),
    }
 
 
def validate(df, schema, max_out_of_range=0.01, max_shift=0.5,
             max_null_increase=0.01):
    """One message per violated expectation. Empty list means the batch is usable."""
    problems = []
    if list(df.columns) != schema["columns"]:
        missing = [c for c in schema["columns"] if c not in df.columns]
        extra = [c for c in df.columns if c not in schema["columns"]]
        if missing:
            problems.append(f"missing columns: {missing}")
        if extra:
            problems.append(f"unexpected columns: {extra}")
        if not missing and not extra:
            problems.append("columns in a different ORDER")
 
    for c in schema["columns"]:
        if c not in df.columns:
            continue
        if str(df[c].dtype) != schema["dtypes"][c]:
            problems.append(f"{c}: dtype {df[c].dtype} != {schema['dtypes'][c]}")
        nulls = float(df[c].isna().mean())
        if nulls > schema["null_rate"][c] + max_null_increase:
            problems.append(f"{c}: null rate {nulls:.4f} "
                            f"(fitted {schema['null_rate'][c]:.4f})")
        col = df[c].dropna()
        if len(col) == 0:
            continue
        out = float(((col < schema["lower"][c]) | (col > schema["upper"][c])).mean())
        if out > max_out_of_range:
            problems.append(f"{c}: {out:.2%} of values outside the fitted range")
        shift = abs(col.mean() - schema["mean"][c]) / max(schema["std"][c], 1e-12)
        if shift > max_shift:
            problems.append(f"{c}: mean shifted {shift:.2f} sd")
    return problems

figure6 of 6 corruptions caught, in 6.65 ms per 1,200-row batchmatplotlib

6.65 ms for 1,200 rows across 25 columns. On a per-request path that is too slow to run on every call; the pattern that works is to validate per batch and, for single-row requests, keep only the cheap checks — names, dtypes, and per-field ranges, which is exactly what a pydanticpydantic model gives you.

Choosing thresholds

The two value checks need numbers, and the numbers determine whether you get useful alarms or noise.

figureThe alarm fires at +0.75, and this shift never costs more than 0.0033matplotlib

And the finding that matters most on this page:

A validation alarm is not a performance prediction. The shift of +0.75 that trips both checks costs essentially nothing (0.8508 against a clean 0.8500), and even +4.0 costs 0.0025. This is the same result as input drift monitoring in Phase 08, where PSI reached 2.0847 with accuracy at 0.9995.

The correct reading is not “these checks are useless”. It is:

• Data validation answers “is this batch like the training data?” — a question about the input, answerable immediately, with no labels.
• Model monitoring answers “is the model still right?” — a question about the output, answerable only when labels arrive.
• The first is a precondition, not a proxy. Run it because a batch that violates the contract may be catastrophic (0.7125) or harmless (0.8475) and you cannot tell which from the model’s output, which will look perfectly normal either way.

Practical policy: hard-fail on schema, dtype and null violations, because those are almost always genuine bugs; warn and investigate on range and shift, because those are often legitimate change in the world.

See it move

The uncomfortable part of this page is that “alarm fired” and “model broke” are close to independent. The sketch makes that a two-by-two you can watch fill up: it cycles through the seven batches measured above, shows which gates catch each one, and plots the resulting accuracy against the alarm state.

The two off-diagonal quadrants are the whole design problem. f00 + 3f00 + 3 fires two gates and costs 0.0025, so a team that pages on every alarm learns to ignore them. The numpy reordering fires nothing and costs 0.0567, so a team that trusts silence ships it. Neither is fixed by tightening a threshold — the first needs a severity policy, and the second needs a contract that survives leaving pandas.

Where the libraries fit

  flowchart TD
  A["A batch arrives"] --> B{"Right columns,
right types?"}
  B -->|"no"| C["HARD FAIL.
Do not predict.
This is a bug upstream."]
  B -->|"yes"| D{"Null rates
as expected?"}
  D -->|"no"| C
  D -->|"yes"| E{"Values in the
fitted ranges?"}
  E -->|"no"| F["WARN. Predict, but flag
the batch and page a human.
Could be units, could be
the world changing."]
  E -->|"yes"| G["Predict"]
  F --> G
  G --> H["Log the validation result
NEXT TO the prediction,
so a later investigation
can join them"]

The choice matters much less than the fields you check. A schema that omits value ranges will pass the unit-change batch no matter which library wrote it.

Pitfalls

Recap

• Of seven corrupted batches, 4 raised a ValueErrorValueError and 3 were served silently; the worst silent one cost 0.1375 accuracy (0.8500 → 0.7125).
• Reordering columns raised as a DataFrame and cost 0.0567 silently as a numpy array.
• A 40-line schema contract caught 6 of 6 corruptions in 6.65 ms per 1,200-row batch.
• Each corruption tripped a different check, which is why you need all five.
• The range and shift checks fired at a shift of +0.75, where accuracy was 0.8508 against a clean 0.8500 — validation is a precondition, not a performance prediction.
• Hard-fail on schema, dtype and nulls; warn and investigate on range and shift.

🧪 Try It Yourself

Exercise 1 – Corrupt a batch six ways

Exercise 2 – Fit the contract

Exercise 3 – Catch all six

Exercise 4 – Find where the alarm starts

Exercise 5 – Serialise the contract with the model

Next

Feature Stores and Training-Serving Skew — the contract above assumes the serving batch is trying to look like training data. The next page is about the case where two different code paths compute the same feature, and quietly disagree.