Every phase before this one asked how well does the model score. This one asks what did it actually do, and the two questions come apart much more often than the first eight phases suggest.

The organising fact of this phase: each of the five standard interpretability tools produces a number that is correct and an impression that is wrong, unless you know the specific way it lies. None of them raise an error. All of them look authoritative in a slide deck.

What this phase covers

  flowchart TD
  M["A fitted model"] --> W{"What do you need to know?"}
  W -->|"Can I trust the mechanism?"| A["1. Why interpretability
audit against domain expectation"]
  W -->|"Which features matter?"| B["2. Permutation importance
on HELD-OUT data"]
  W -->|"Which direction?"| C["3. PDP + ICE
never PDP alone"]
  W -->|"Why this row?"| D["4. SHAP
sums exactly to the prediction"]
  W -->|"Who is affected?"| E["5. Group metrics
at the deployed threshold"]
  B --> F["Averaging local explanations
gives the global picture"]
  D --> F
  A --> G{"Defensible to a
domain expert?"}
  C --> G
  E --> G
  F --> G
  G -->|"no"| H["Do not deploy"]
  G -->|"yes"| I["Deploy, and keep measuring"]

The five measured traps

Every number below is produced by code on the page it belongs to, and every one of them is the default behaviour of a standard tool.

A 0.9642 score that meant nothing. A forest trained at one hospital scored 0.9642 on a clean held-out split and 0.4998 — a coin flip — at a second hospital. It had learned scanner_idscanner_id, whose correlation with the diagnosis was 0.9460 at the first site and −0.0007 at the second. Permutation importance on the first site’s own held-out data already said so: scanner_idscanner_id +0.4623, the honest clinical marker +0.0000 ± 0.0000.

Impurity importance ranked pure noise first. feature_importances_feature_importances_ gave a column of Gaussian noise 0.3853 of the total credit — rank 1 of 5, above all three genuinely informative features — because impurity importance is computed on training data and rewards cardinality. Permutation on the training set still inflated it to +0.1796. Only permutation on held-out data put it at +0.0134.

A partial dependence plot that said “no effect” about a huge effect. The PDP’s total range was 0.8011 across the whole sweep. The per-row ICE curves ran from a slope of −9.0235 to +8.5737. The PDP is the average of those, and the average of two opposite effects is approximately nothing — a 21.97× understatement.

SHAP is exact, and exactness costs $2^p$. Computed from the definition, the contributions summed to the prediction with an error of 0.00e+00 — base value −0.2102 plus contributions +1.3965 equals the actual prediction +1.1863. That guarantee requires evaluating every coalition: 16 subsets at 4 features, over a million at 20. Every production library approximates, and the approximation is where the caveats live.

Fairness gaps without a protected attribute. The model was never given the group column and the data contained no measurement bias, yet the selection-rate gap was 0.6537. Closing it required dropping one group’s threshold to 0.0050, which took overall accuracy from 0.7994 to 0.6103 and that group’s precision from 0.5551 to 0.2649.

The result that reorganises how you read every chart

Global and local explanations answer different questions, and the relationship between them runs one way only. On a four-feature linear model whose true coefficients are $3, -2, 1, 0$:

f2_weakf2_weak is a distant third globally and the deciding factor for more than one applicant in ten. On row 102 it contributes −2.5453 of a −2.8302 total move away from the base value, while the top-ranked feature contributes +0.2500.

Averaging local explanations reconstructs the global picture; no amount of global information recovers a local one. Which means: if the question came from a person about their own case, a global chart is not an answer to it — it is a different, true statement about something else.

Before you start

• Bagging & Random Forests — feature_importances_feature_importances_ is a forest attribute, and page 2 is largely about why you should not read it.
• K-Fold Cross-Validation — every trap here turns on the difference between training and held-out data.
• Monitoring Model Drift — the shortcut failure on page 1 is a drift failure that no input-distribution monitor would flag.
• Precision, Recall, and F1-Score — page 5 is those four cells computed per group, and the arithmetic that links them.

No new libraries. There is no shapshap and no fairlearnfairlearn in this phase: Shapley values are computed exactly, by brute force over every coalition, and each fairness metric is three lines of numpy. You end up reading definitions instead of API surfaces.

What you’ll be able to do afterwards

1. Audit a model’s reliance on each feature against what a domain expert expected, and stop a deployment on the result.
2. Explain why feature_importances_feature_importances_ put an ID column first, and what to compute instead.
3. Read a PDP without being fooled by it, and know when the ICE curves contradict it.
4. Implement Shapley values from the definition, and verify local accuracy to floating-point error.
5. Explain a single prediction in terms that sum exactly to that prediction.
6. Compute all four group fairness criteria at the threshold you actually deploy.
7. Say precisely why they cannot be satisfied together, using Chouldechova’s identity.
8. Tell the difference between an explanation of the model and a claim about the world.

How long it takes

Practice project

Write the model card nobody can argue with.

Take the best model you built in any earlier phase — ideally one with a real dataset and at least a dozen features.

Part 1 — Establish the transparent baseline. Fit logistic regression or a depth-3 tree alongside your model. Record both scores and both “numbers a human must read” counts. If the gap is under 0.01, say so in writing; you may have just saved yourself the rest of this project.

Part 2 — Audit the mechanism. Permutation importance on held-out data, with n_repeatsn_repeats at least 30 and standard deviations printed. Write down, before looking, which three features you expect to lead. Then compare. Anything important that you did not expect is the finding — chase it until you can explain it.

Part 3 — Check the direction. PDP with method="brute"method="brute" plus ICE for your top three features. Report the PDP range and the spread of ICE slopes side by side. Where they disagree, find the interacting feature.

Part 4 — Explain three individual rows. One confident positive, one confident negative, one right at the decision boundary. Verify additivity to at least 1e-10 on each. Write each explanation as a sentence you would be willing to send to the person that row describes.

Part 5 — Audit for group differences. Pick a group column — if your data has no protected attribute, use any categorical column and treat it as one. Report base rate, selection rate, TPR, FPR and precision per group with counts, at the threshold you would deploy. Then choose one criterion, justify the choice in two sentences by naming the costly error, and report what enforcing it costs.

Part 6 — Write the card. Two pages, no more: intended use, out-of-scope use, the features it leans on, direction of each top effect, the group table, the criterion you chose and why, and the things you know are still wrong with it.

Part 2 is where the surprises live. If your audit produced no surprise at all, check that your expectations were written down first — it is remarkably easy to rediscover what you already saw.

Next

Start with Why Interpretability Matters — the 0.9642 model that was worth nothing, and the single permutation that would have caught it before it shipped.