What you’ll learn

• how a bag of words turns 6,000 short tickets into a 2,447 × 4,200 matrix that is 99.17% zeros
• what idf actually computes, and why the loudest weights go to the least useful tokens
• why sparse linear models win on text: 0.8733 for Naive Bayes against 0.8100 for boosting on 120 SVD components
• that min_df=50min_df=50 deletes 92% of the vocabulary and improves accuracy to 0.8711
• how a template artefact becomes a 5.46 coefficient, and costs 15 accuracy points when the template changes
• when character n-grams beat word n-grams: 0.8611 against 0.8289 on typo’d text

The corpus

Real support tickets cannot be published, so this page uses a synthetic corpus generated from a known process: 6,000 tickets, three queues, about 22 words each. Filler words, a handful of topical words, and a long tail of case identifiers.

TOPICS = {
    "billing":   "invoice charged refund payment card ... late lost error".split(),
    "shipping":  "parcel delivery courier tracking ... late lost refund".split(),
    "technical": "login password reset error crash ... charge missing".split(),
}

TOPICS = {
    "billing":   "invoice charged refund payment card ... late lost error".split(),
    "shipping":  "parcel delivery courier tracking ... late lost refund".split(),
    "technical": "login password reset error crash ... charge missing".split(),
}

Three properties are deliberate, and each one shows up in a measurement later:

• The topic vocabularies overlap. refundrefund belongs to billing and shipping; errorerror to billing and technical; latelate and lostlost to billing and shipping. This is the irreducible part.
• 6% of the labels are wrong — tickets filed in the wrong queue, as happens.
• 22% of the tokens are case identifiers drawn from a Zipf distribution over 4,000 possible values, which is what makes the matrix wide.

One caveat to state up front: the generator shuffles the words in each document, so word order carries no information by construction. That makes this corpus a clean test of bag-of-words methods and a useless test of anything that depends on sequence — a limitation the n-gram section returns to.

The matrix

figure6,000 tickets of about 22 words produce a matrix that is 99.17% zerosmatplotlib

What idf does

Term frequency alone gives every word its raw count. Inverse document frequency down-weights words that appear everywhere. scikit-learn’s smoothed version is

after which each row is L2-normalised, so document length stops mattering.

figure43% of the vocabulary appears in exactly one document, and idf hands those tokens the largest weightsmatplotlib

Two things are worth noticing.

The most frequent token is an identifier. case00000case00000 is the head of the Zipf distribution over case numbers and appears in half the documents, so idf treats it as a stopword — correctly, by accident.

idf does not know what matters. errorerror gets the highest weight of the seven above, and it is one of the overlapping words that cannot separate billing from technical. idf measures how unusual a token is, not how diagnostic. Supervised weighting is what the classifier’s coefficients do.

See it move

The idf formula is four operations, and the intuition arrives faster by moving it than by reading it. Toggle which of the eight documents contain the word.

Two boundary cases are worth clicking to: a word in all eight documents has idf exactly 1.0000, so tf-idf reduces to term frequency; a word in none of them has no column at all, which is what happens to every token that appears for the first time in production.

Models

figureNaive Bayes on tf-idf beats a 120-component boosting pipeline by 0.06matplotlib

Four things this ordering tells you:

tf-idf is worth 0.0273 over raw counts for the same classifier — a real but modest gain, entirely from down-weighting the identifier tokens that dominate the counts.

Naive Bayes is not a toy. Its independence assumption is false, and it still wins here, in milliseconds, with no hyperparameters. On short documents with sparse features it is the correct baseline and frequently the answer.

Trees are the wrong shape for text. Each split tests one token; discriminating three classes on 1.98 topical words per document needs many weak pieces of evidence added together, which is what a linear model does by construction and what a tree does only with enormous depth.

Dimensionality reduction did not help. SVD-120 + logistic (0.8700) is statistically level with sparse logistic (0.8656) while discarding 2,327 columns and adding a fitted transform to maintain. Reach for it when you need dense vectors for something else, not for accuracy.

The confusion matrix shows where the remaining 13% goes:

The errors are almost perfectly symmetric — 48 and 36 between billing and shipping, 40 and 32 between billing and technical. Symmetric confusion is the signature of genuinely ambiguous documents (a refundrefund for a damaged parceldamaged parcel belongs to both queues) rather than a model that has learned one class badly.

Most of the vocabulary is dead weight

figure2,447 features and 207 features score within 0.0056matplotlib

min_dfmin_df is the cheapest regularisation available on text, and it is a statement about generalisation rather than about memory: a token that appears in three training documents cannot support a reliable coefficient, and keeping it invites the model to memorise those three.

The companion knob is max_dfmax_df, which drops tokens appearing in too many documents. On this corpus it does little, because idf already handles them — but on corpora with boilerplate (legal disclaimers, email footers) max_df=0.5max_df=0.5 removes a lot of noise in one line.

Two failures that clean cross-validation cannot show you

figureTwo failure modes that no cross-validation on clean, same-template data can show youmatplotlib

The template artefact

30% of billing tickets carry the token ref_billref_bill — a reference-number prefix from the ticketing template, with no meaning whatsoever. The model finds it immediately:

ref_billref_bill outweighs the best genuine billing word by 1.7×, and on the 193 test tickets that carry it the model is right 94.82% of the time against 85.56% elsewhere. Everything looks excellent.

Now migrate the ticketing system so the prefix disappears, and score the same model on the same tickets with that one token removed: 0.7927. A 15.6-point drop, concentrated entirely in the segment that used to be the model’s best.

This is not leakage in the train/test sense — the token is genuinely present at prediction time, and every split reproduces it. It is a dependency on an artefact, and the only defence is reading the coefficients and asking whether each top token is something a human would use to make the same decision. Which is exactly the audit from Phase 09.

Typos, and character n-grams

Word-level features are exact-match: deliverydelivery and delivrydelivry are unrelated columns. Delete one letter from 25% of the words longer than four characters — a mild simulation of real user text — and word tf-idf falls from 0.8656 to 0.8289. Character n-grams, which see delivdeliv in both spellings, fall only from 0.8700 to 0.8611.

TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5), min_df=2)
# 7,637 features on this corpus; robust to typos, agglutination and casing

TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5), min_df=2)
# 7,637 features on this corpus; robust to typos, agglutination and casing

The cost is a wider matrix (7,637 features against 2,447) and features nobody can read. The usual production answer is a FeatureUnionFeatureUnion of both: word n-grams for interpretability, character n-grams for robustness.

On n-grams

Bigrams add 16,239 columns and 0.0000 of accuracy here — and that result is a property of this corpus, not of text in general. The generator shuffles words, so there are no phrases to find. On real text, bigrams typically buy one to three points on sentiment (“not good”) and almost nothing on topic classification. The lesson is procedural: measure the gain on your corpus before paying for a tenfold wider matrix.

The pipeline

  flowchart TD
  A["Raw documents"] --> B["TfidfVectorizer
lowercase, min_df=2..10"]
  B --> C["Linear model or
Naive Bayes"]
  C --> D{"Beats the majority
class by enough?"}
  D -->|"no"| E["Look at the confusion matrix.
Symmetric = ambiguous labels.
Asymmetric = a broken class."]
  D -->|"yes"| F["Read the top coefficients
per class"]
  F --> G{"Would a human use
these tokens?"}
  G -->|"no — templates,
ids, footers"| H["Remove them and re-measure.
That number is the real one."]
  G -->|"yes"| I["Check robustness:
typos, new vocabulary,
a different template"]
  I --> J["Ship the sparse pipeline.
Pickle the vectorizer WITH
the model."]

The last box is the most common production bug on this page’s subject: a TfidfVectorizerTfidfVectorizer is a fitted object holding the vocabulary and the idf vector. Save the whole PipelinePipeline, never the classifier alone — the same failure, with the same silence, as dropping a scaler.

Pitfalls

Recap

• 6,000 tickets, 22 words each, produced a 2,447-column matrix that is 99.17% zeros, with 43.0% of tokens appearing in exactly one document.
• idf is $\log((1+n)/(1+\mathrm{df})) + 1$: case00000case00000 got 1.6853 and errorerror 3.8920 — it measures rarity, not relevance.
• Multinomial Naive Bayes 0.8733, logistic on tf-idf 0.8656, logistic on raw counts 0.8383, SVD-120 + boosting 0.8100.
• min_df=50min_df=50 cut the vocabulary by 92% and raised accuracy to 0.8711.
• A meaningless template token earned the model’s largest coefficient (+5.459) and cost 15.6 points on the affected tickets when it disappeared.
• Under mild typos, word features lost 0.0367 and character 3–5 grams 0.0089.
• Bigrams added 16,239 columns and nothing — on this corpus, whose words are shuffled by construction.

🧪 Try It Yourself

Exercise 1 – Generate the corpus

Exercise 2 – Vectorize, and look at the matrix

Exercise 3 – Fit three models and read the confusion matrix

Exercise 4 – Delete 92% of the vocabulary

Exercise 5 – Break the template

Next

Recommender Systems from Scratch — from a sparse document-term matrix to a sparser user-item matrix, where 99.17% zeros would count as dense and the missing entries are the thing you are trying to predict.