The decision

A payments team screens transactions. Flagged ones go to a human reviewer; unflagged ones settle.

The deliverable is not a model. It is a threshold, an alert volume, and an expected loss per transaction — three numbers the business can act on, plus the evidence that the model reads what a fraud analyst would expect it to read.

This capstone composes five earlier pages: the metric choice, the cost-derived threshold, the importance audit, the segment check and the schema contract.

Step 1 — split three ways, and pick the metric first

# 50% to fit, 20% to choose operating points, 30% to report once
X_fit, X_rest, y_fit, y_rest, a_fit, a_rest = train_test_split(
    X, y, amount, test_size=0.5, random_state=0, stratify=y)
X_val, X_te, y_val, y_te, a_val, a_te = train_test_split(
    X_rest, y_rest, a_rest, test_size=0.6, random_state=0, stratify=y_rest)
# fit 10,000 / 44 frauds   val 4,000 / 17   test 6,000 / 26

# 50% to fit, 20% to choose operating points, 30% to report once
X_fit, X_rest, y_fit, y_rest, a_fit, a_rest = train_test_split(
    X, y, amount, test_size=0.5, random_state=0, stratify=y)
X_val, X_te, y_val, y_te, a_val, a_te = train_test_split(
    X_rest, y_rest, a_rest, test_size=0.6, random_state=0, stratify=y_rest)
# fit 10,000 / 44 frauds   val 4,000 / 17   test 6,000 / 26

Average precision is the headline because the review workload scales with alerts, and Brier is on the list because the next step consumes probabilities rather than a ranking.

Step 2 — turn the costs into a policy

figureCheapest policy: cost-optimal t* = 0.0099 at EUR 3,410 — EUR 0.5683 per transactionmatplotlib

† The last row is priced against the transaction amount rather than a flat EUR 500, so its total is not comparable with the rows above it. Within that costing it is the cheapest policy available.

Three numbers to carry into the meeting:

• Expected loss per transaction: EUR 0.5683 at $t^*$, against EUR 2.1667 doing nothing. That is the business case, and it is a per-unit number so it scales.
• 304 alerts per 6,000 transactions — 5.1% of volume. At six analysts × 50 alerts that is a hard constraint, and the model does not get to ignore it.
• The capacity policy costs EUR 1,255 more than the unconstrained optimum. That difference is the value of hiring the seventh analyst, expressed in the same units as their salary.

See it move

The four policies in that table are four points on one curve. The sketch is that curve, measured on the same 6,000 test transactions at 43 thresholds — drag the threshold and read the alert count, the misses and the bill. The analyst-capacity line at 50 alerts is drawn where operations actually sits.

One honest wrinkle the curve exposes: on this particular test set the cheapest threshold measured is 0.00346 at EUR 2,675, not $t^*$ at EUR 3,410. That is not evidence that the formula is wrong — with 26 frauds, the realised cost of any threshold has a standard error of hundreds of euros, and the empirical argmin is itself a selected number with exactly the winner’s curse built in. $t^*$ minimises expected cost from the cost ratio alone and needs no tuning; the empirical minimum minimises this sample and would move on the next one. Ship the formula, and note the flat bottom: anything from roughly 0.003 to 0.04 costs under EUR 4,000, so the decision is robust even though the argmin is not.

Step 3 — audit what the model reads

Permutation importance on the test set, scored by average precision, 20 repeats:

Two checks pass here. The five features a fraud analyst would name are the five that matter, in a plausible order. And the deliberately uninformative column scores −0.0011 ± 0.0052 — indistinguishable from zero, which is what a column of noise should score. Had noisenoise ranked third, the correct response would be to stop and find out why, exactly as on the shortcut page.

Step 4 — check the segments before shipping

The model never sees the channel a transaction arrived through. Slice the results by it anyway:

Recall ranges from 0.7500 to 0.9231 across channels. Before treating that as a finding, note the denominators: 4 frauds in the phone segment. One fraud either way moves that recall by 0.25, so the honest statement is “no measurable difference between channels, and the phone segment is too small to tell” — the same discipline as any fairness slice. What you can commit to is monitoring it monthly, when the counts accumulate.

Step 5 — the engineering that makes it a system

CONFIG = {"c_fp": 5.0, "c_fn": 500.0, "model_version": "fraud-1.2.0"}
 
 
def screen(batch, artefact, config):
    """One batch: validate, score, threshold, and log everything an audit needs."""
    problems = validate(batch, artefact["schema"])
    hard = [p for p in problems if "column" in p or "null rate" in p]
    if hard:
        raise ValueError(f"unusable batch: {hard}")          # do not predict
 
    proba = artefact["model"].predict_proba(batch)[:, 1]
    threshold = config["c_fp"] / (config["c_fp"] + config["c_fn"])
    flagged = proba >= threshold
 
    return {
        "model_version": config["model_version"],
        "schema_id": artefact["schema_id"],
        "threshold": threshold,
        "rows": len(batch),
        "alerts": int(flagged.sum()),
        "alert_rate": round(float(flagged.mean()), 4),
        "mean_score": round(float(proba.mean()), 6),
        "warnings": problems,                                # statistics, not structure
        "scores": proba,                                     # logged per row
    }

CONFIG = {"c_fp": 5.0, "c_fn": 500.0, "model_version": "fraud-1.2.0"}
 
 
def screen(batch, artefact, config):
    """One batch: validate, score, threshold, and log everything an audit needs."""
    problems = validate(batch, artefact["schema"])
    hard = [p for p in problems if "column" in p or "null rate" in p]
    if hard:
        raise ValueError(f"unusable batch: {hard}")          # do not predict
 
    proba = artefact["model"].predict_proba(batch)[:, 1]
    threshold = config["c_fp"] / (config["c_fp"] + config["c_fn"])
    flagged = proba >= threshold
 
    return {
        "model_version": config["model_version"],
        "schema_id": artefact["schema_id"],
        "threshold": threshold,
        "rows": len(batch),
        "alerts": int(flagged.sum()),
        "alert_rate": round(float(flagged.mean()), 4),
        "mean_score": round(float(proba.mean()), 6),
        "warnings": problems,                                # statistics, not structure
        "scores": proba,                                     # logged per row
    }

Four things this earns:

• Hard-fail on structure, warn on statistics — the policy from the validation page.
• alert_ratealert_rate is the operational canary. It is available immediately, needs no labels, and a jump from 5.1% to 12% means something changed today, whatever the labels eventually say.
• mean_scoremean_score catches recalibration. The model’s mean predicted probability should track the base rate (0.0043 here). A drift in it invalidates the threshold, because $t^*$ assumes calibration.
• The threshold is derived from config, never hard-coded. When finance revises the fraud cost, one number changes and nothing retrains.

The test suite, from Testing ML Code: a metric floor (average precision ≥ 0.30 on the fixed split), memorisation of 24 rows, a directional expectation on velocityvelocity, a structural assertion that the scaler is inside the PipelinePipeline, and a schema round-trip.

The system those pieces add up to, with the two feedback paths that make it survive contact with production:

  flowchart TD
  B["incoming batch of transactions"] --> V{{"validate against
the schema artefact"}}
  V -->|"structural problem:
missing column, null rate"| STOP["raise. do not predict.
a wrong answer is worse
than no answer"]
  V -->|"statistical warning:
range, mean shift"| LOGW["log the warning
and continue"]
  V -->|"clean"| SCORE
  LOGW --> SCORE["model.predict_proba
fraud-1.2.0"]
  CFG[("config:
c_fp = 5, c_fn = 500")] -->|"t* = c_fp / (c_fp + c_fn)"| THR{{"threshold at 0.0099"}}
  SCORE --> THR
  THR -->|"score >= t*"| Q["alert queue,
ranked by score"]
  THR -->|"score < t*"| PASS["let through"]
  Q --> CAP{{"analyst capacity:
50 reviews per shift"}}
  CAP -->|"within capacity"| REV["human review"]
  CAP -->|"queue longer than capacity"| SPILL["review top-ranked only.
the overflow is an
accepted, measured loss"]
  SCORE --> MON["log alert_rate and mean_score
-- no labels needed"]
  MON -->|"alert_rate jumps 5.1% -> 12%,
or mean_score leaves 0.0043"| INV["investigate today:
upstream change, or
the threshold is stale"]
  REV --> LAB["confirmed labels,
weeks later"]
  SPILL --> LAB
  LAB --> RETRAIN["retrain and re-audit;
recompute t* if finance
revises the costs"]
  RETRAIN -.-> SCORE
  INV -.-> CFG

Note what is not in the model: the capacity constraint, the cost ratio and the review workflow all live in configuration and operations. That separation is what lets finance change c_fnc_fn on a Tuesday without a retraining run, and it is the difference between a model and a system.

What this project does not solve

• The 26 test frauds. Every number here has a wide confidence interval; the bootstrap on this data spanned 0.37 of average precision. Treat model comparisons as inconclusive until far more positives accumulate.
• Label delay. Chargebacks arrive weeks later, so today’s recall is unknowable today. The alert rate and score distribution are the only same-day signals.
• Adversarial drift. Fraud patterns react to the screening policy. Nothing on this page models an adversary who learns your threshold.
• The review process. A 93%-false-alert queue has a human cost — fatigue, inconsistency — that the EUR 5 does not capture.

Recap

• 6,000 test transactions, 26 frauds (0.43%); average precision 0.4387 against a 0.0043 chance line.
• The default threshold cost EUR 12,005 against EUR 13,000 for doing nothing.
• $t^* = C_{FP}/(C_{FP}+C_{FN}) = 0.0099$ cost EUR 3,410: 304 alerts, 4 missed, precision 0.0724, recall 0.8462.
• Expected loss per transaction: EUR 0.5683 against EUR 2.1667.
• A 50-alert capacity limit cost EUR 4,665 — EUR 1,255 more than the unconstrained optimum.
• The audit passed: the five plausible features led, and noisenoise scored −0.0011 ± 0.0052.
• Channel recall ranged 0.7500–0.9231 on 4–13 frauds per segment: no measurable difference.

🧪 Try It Yourself

Exercise 1 – Build the screening dataset and split it three ways

Exercise 2 – Report the right metric

Exercise 3 – Price four policies

Exercise 4 – Audit what the model reads

Exercise 5 – Slice by a segment the model never saw

Next

Capstone 2 - Churn with Point-in-Time Features — the same decision structure on an event log, where the hard part is not the model but making sure the features could have existed when the prediction was made.