The decision

Support tickets arrive as free text and have to reach one of three queues: billing, shipping or technical. Today a person reads each one.

The model is allowed a third option: abstain. Route the ticket automatically when confident, hand it to a person otherwise. So the deliverable is not an accuracy — it is a confidence threshold, a coverage figure, and the accuracy on the covered portion.

This capstone composes the text pipeline, the coefficient audit and the cost reasoning from Capstone 1.

Step 1 — the classifier, and its ceiling

TfidfVectorizer(min_df=2)TfidfVectorizer(min_df=2) plus multinomial logistic regression on 4,200 training tickets:

The 6% label noise matters more than any modelling choice here: a ticket whose recorded queue is wrong cannot be predicted correctly, so no threshold and no model can reach 1.0. Knowing that number is what makes the next step honest.

Step 2 — let the model abstain

figureTriage by hand costs EUR 2; a misrouted ticket costs EUR 12. Neither extreme is optimal.matplotlib

Four things to take from this table.

Abstention is worth more than a better model. The classifier’s accuracy is 0.8672 and nothing about it changed. Allowing it to decline the hard 26% took cost from EUR 2,868 to EUR 1,664 — a saving larger than any plausible modelling improvement on this corpus.

The operating point is a cost question, not an accuracy question. At 0.50 the model covers 84.00% at 0.9392; at 0.60 it covers 73.78% at 0.9548. Their costs are EUR 1,680 and EUR 1,664 — a tie. Choose between them on operational grounds: how many tickets your team can absorb, and whether a misroute has a reputational cost the EUR 12 does not capture.

Accuracy on the routed portion plateaus at about 0.96. Beyond a 0.70 threshold, raising the bar buys almost nothing: 0.9610 at 61% coverage, 0.9611 at 27% coverage. The remaining errors are confident errors, which is what 6% label noise produces — the model is confidently right and the recorded label is wrong. No amount of abstention fixes a mislabelled ticket.

The cost curve is flat between 50% and 84% coverage. Costs are EUR 1,664–1,910 across that whole band, which is good news operationally: you can set the threshold from staffing capacity and lose very little.

See it move

The table has eight rows; the decision has two independent inputs. The sketch lets you move the confidence threshold along the measured curve and re-price the two costs, so you can see the operating point move for reasons that have nothing to do with the model.

Two experiments worth running in it, both leaving the classifier untouched:

Make misroutes more expensive and the model is trusted with less; make human triage more expensive and it is trusted with more — coverage swings from 61% to 96% across those rows. Same accuracy, same confidence scores, four different systems. The operating point is an economic quantity that happens to be implemented as a number compared against predict_probapredict_proba.

Step 3 — audit the tokens before trusting the routing

The text page found that this corpus contains ref_billref_bill, a template artefact carried by 30% of billing tickets. Its coefficient is +5.459 — larger than any real billing word — and the model’s accuracy on those tickets drops from 0.9482 to 0.7927 if the ticketing system stops emitting it.

For a triage system that is a specific, dated risk: the next migration of the ticket template silently degrades one queue’s routing. Two mitigations, both cheap:

• Put the artefact on a watchlist. Log the share of tickets containing ref_billref_bill per day. When it changes, the routing accuracy for billing is about to change too.
• Train a variant without it and keep the measurement. Removing ref_billref_bill from the vocabulary and refitting scores 0.8528 against 0.8672 — a cost of 1.4 points of accuracy. That prices the dependency exactly: the artefact buys 1.4 points today and owes you one migration-shaped outage.

Step 4 — the routing service

CONFIG = {"min_confidence": 0.60, "human_cost": 2.0, "misroute_cost": 12.0,
          "model_version": "triage-2.1.0"}
 
 
def triage(ticket_text, artefact, config):
    """Route automatically when confident, otherwise send to a human."""
    features = artefact["vectorizer"].transform([ticket_text])
    proba = artefact["model"].predict_proba(features)[0]
    queue = artefact["classes"][int(proba.argmax())]
    confidence = float(proba.max())
 
    decision = "auto" if confidence >= config["min_confidence"] else "human"
    return {
        "queue": queue if decision == "auto" else None,
        "decision": decision,
        "confidence": round(confidence, 4),
        "runner_up": artefact["classes"][int(proba.argsort()[-2])],
        "margin": round(float(np.diff(np.sort(proba)[-2:])[0]), 4),
        "unknown_token_share": artefact["unknown_share"](ticket_text),
        "model_version": config["model_version"],
    }

CONFIG = {"min_confidence": 0.60, "human_cost": 2.0, "misroute_cost": 12.0,
          "model_version": "triage-2.1.0"}
 
 
def triage(ticket_text, artefact, config):
    """Route automatically when confident, otherwise send to a human."""
    features = artefact["vectorizer"].transform([ticket_text])
    proba = artefact["model"].predict_proba(features)[0]
    queue = artefact["classes"][int(proba.argmax())]
    confidence = float(proba.max())
 
    decision = "auto" if confidence >= config["min_confidence"] else "human"
    return {
        "queue": queue if decision == "auto" else None,
        "decision": decision,
        "confidence": round(confidence, 4),
        "runner_up": artefact["classes"][int(proba.argsort()[-2])],
        "margin": round(float(np.diff(np.sort(proba)[-2:])[0]), 4),
        "unknown_token_share": artefact["unknown_share"](ticket_text),
        "model_version": config["model_version"],
    }

Two fields earn their place beyond the obvious ones. marginmargin — the gap between the top two probabilities — is a better abstention signal than the maximum alone when the classes are unbalanced, and it is what a reviewer wants to see next to a borderline decision. unknown_token_shareunknown_token_share is the fraction of the ticket’s words that were not in the training vocabulary: it rises when the product launches a new feature, and it rises before accuracy falls, which makes it the earliest available warning on a text model.

One ticket’s path through the service, with the three signals that get logged whether or not a human ever sees it:

  flowchart TD
  T["a ticket arrives"] --> VEC["TfidfVectorizer.transform
fitted vocabulary, min_df=2"]
  VEC --> UNK["unknown_token_share:
words outside the vocabulary"]
  VEC --> P["predict_proba over the queues"]
  P --> C{"max probability >= 0.60?"}
  C -->|"yes -- 73.78% of tickets"| AUTO["route automatically.
0.9548 accurate here."]
  C -->|"no -- 26.22%"| HUM["send to a human,
with the top two queues
and the margin attached"]
  P --> MARG["margin: gap between
the top two probabilities"]
  AUTO --> LOG["log queue, confidence, margin,
unknown_token_share, model_version"]
  HUM --> LOG
  LOG --> W1{{"ref_bill share per day"}}
  LOG --> W2{{"unknown_token_share
trending up"}}
  LOG --> W3{{"coverage drifting from 73.78%"}}
  W1 -->|"template migration"| ACT["billing queue accuracy is
about to drop 0.9482 -> 0.7927.
Switch to the variant that
costs 1.4 points and does
not depend on it."]
  W2 -->|"new product vocabulary"| RETRAIN["refit the vectorizer.
This rises BEFORE accuracy falls."]
  W3 -->|"more abstentions than staffed for"| OPS["re-derive the threshold from
current costs and capacity"]
  HUM --> CORR["the human's chosen queue
is a fresh, correct label"]
  CORR --> RETRAIN

The bottom edge is the part that compounds. Every abstention produces a human decision on exactly the tickets the model found hardest, which is the highest-value training data available — so a selective classifier is also a labelling pipeline, provided somebody wires the corrections back.

Step 5 — what to monitor

The re-triage rate is the only ground truth, and it arrives late — so the first four are what you page on. A drop in coverage from 73.78% to 55% means the model is seeing tickets it does not recognise, and it means it today, not after the labels arrive.

What this project does not solve

• The 6% label noise is the ceiling and nobody has measured it in production. Estimating it — by double-labelling a sample of tickets — would tell you whether 0.96 is the ceiling or 0.99 is.
• Word order carries no information in this corpus by construction, so the finding that bigrams add nothing does not transfer to real tickets.
• Three queues, single-label. Real triage is hierarchical (queue → team → skill) and often multi-label, and abstention interacts with that in ways not modelled here.
• The EUR 12 is an average. A misrouted billing dispute and a misrouted password reset do not cost the same, and per-class costs would give per-class thresholds — exactly like the per-row thresholds in Phase 10.

Recap

• 1,800 test tickets, three queues, overall accuracy 0.8672, with 6% of labels wrong by construction.
• Routing everything costs EUR 2,868; routing nothing costs EUR 3,600.
• Routing the 73.78% above 0.60 confidence costs EUR 1,664 at 0.9548 accuracy — a 54% saving against all-human.
• Selective accuracy plateaus near 0.96 above a 0.70 threshold: the remaining errors are confident ones caused by label noise.
• The cost curve is flat between 50% and 84% coverage, so the threshold can follow staffing capacity.
• The ref_billref_bill template token has coefficient +5.459 and costs 0.9482 → 0.7927 on affected tickets when it disappears; it belongs on a watchlist.

🧪 Try It Yourself

Exercise 1 – Fit the triage classifier

Exercise 2 – Build the risk–coverage curve

Exercise 3 – Price each operating point

Exercise 4 – Find the coverage at a required accuracy

Exercise 5 – Add the abstention signals a reviewer needs

Next

Capstone 5 - A Recommender with an Honest Evaluation — the last project, where the hardest engineering is not the model but the measurement.