Handling Text & Categorical Attributes

What you’ll learn

• why OrdinalEncoder makes a claim about your data, and when that claim is false
• one-hot encoding, and the geometric reason it is the safe default
• the memory arithmetic behind sparse matrices — 20,640 × 5 stored in 165 KB or 826 KB
• the dummy variable trap, and when drop="first" matters and when it does not
• what happens when an unseen category arrives in production
• four alternatives for high-cardinality columns, with the cost of each

Intuition

Almost no model accepts the string "NEAR BAY". Encoding turns text into numbers, and the encoding you pick is not a formatting decision — it tells the model what it is allowed to believe about those categories.

Map five categories to 0–4 and you have asserted that they lie on a line, evenly spaced, in that order. Sometimes true (small < medium < large). Here, false: ISLAND is not “between” INLAND and NEAR BAY in any sense, yet a linear model will treat the numbers exactly that way.

diagram Diagram mermaid

  flowchart LR
  T["Text column"] --> Q{"Do the categories
have a real order?"}
  Q -->|yes| O["OrdinalEncoder
1 column, ordering preserved"]
  Q -->|no| C{"How many
distinct values?"}
  C -->|"under ~50"| H["OneHotEncoder
one column each"]
  C -->|"hundreds or more"| A["Target / frequency /
hashing / embedding"]

The one text column

inspect.py

print(housing["ocean_proximity"].value_counts())
 
# <1H OCEAN     9136
# INLAND        6551
# NEAR OCEAN    2658
# NEAR BAY      2290
# ISLAND           5

figure Five categories spanning three orders of magnitude matplotlib

ISLAND appears in 5 districts out of 20,640. A random split has a real chance of putting zero of them in the training set — and then the encoder has never seen the category it meets at test time.

Ordinal encoding

ordinal.py

from sklearn.preprocessing import OrdinalEncoder
 
encoder = OrdinalEncoder()
encoded = encoder.fit_transform(housing[["ocean_proximity"]])
 
print(encoder.categories_)
# [array(['<1H OCEAN', 'INLAND', 'ISLAND', 'NEAR BAY', 'NEAR OCEAN'], dtype=object)]
print(encoded[:5].ravel())      # [3. 3. 3. 3. 3.]

Categories are assigned integers in alphabetical order, which is where the trouble starts. ISLAND becomes 2 purely because “I” sorts between “I” and “N” — and a linear model then reads that as: an island district is numerically between an inland one and a bay one, and twice as far from <1H OCEAN as INLAND is.

figure What each encoding asserts matplotlib

Ordinal places the categories on a line with distances. One-hot places them on the corners of a simplex, where every pair is exactly the same distance apart — the honest representation when no order exists.

Geometrically: under one-hot every pair of categories is $\sqrt{2}$ apart, so no ordering and no spacing is implied. Under ordinal the distances are 1, 2, 3 and 4, and the model uses them.

See it move

Distances are the whole argument, so the sketch computes them. Hover a category to select it; the bars show how far every other category sits from your selection under each encoding.

sketch Distances the two encodings assert p5.js Pause Restart

Five ocean_proximity categories under ordinal and one-hot encoding. Selecting a category shows its distance to every other: the ordinal bars are unequal and imply an ordering, the one-hot bars are all the square root of two. Hover to change the selection.

The amber bars are the encoder’s uninvited claim. Nothing in the data says ISLAND sits between INLAND and NEAR BAY; alphabetical order put it there, and a linear model reads it as a distance. The green bars assert only “different”, which is all that is true.

When ordinal is correct, pass the order explicitly

For genuinely ordered categories, never rely on alphabetical sorting:

python

OrdinalEncoder(categories=[["small", "medium", "large", "x-large"]])

Alphabetically that would be large, medium, small, x-large — an order that is not merely arbitrary but actively wrong.

Tree models are the exception. A tree splits on thresholds, so ordinal <= 1.5 can isolate any group of categories the ordering happens to make contiguous. Ordinal encoding for trees is often fine and much cheaper than one-hot, which is why gradient boosting libraries offer native categorical support built on exactly this idea.

One-hot encoding

onehot.py

from sklearn.preprocessing import OneHotEncoder
 
encoder = OneHotEncoder()
one_hot = encoder.fit_transform(housing[["ocean_proximity"]])
 
print(one_hot)
# <Compressed Sparse Row sparse matrix of dtype 'float64'
#   with 20640 stored elements and shape (20640, 5)>
 
print(one_hot.shape)                       # (20640, 5)
print(encoder.get_feature_names_out())
# ['ocean_proximity_<1H OCEAN' ... 'ocean_proximity_NEAR OCEAN']
 
dense = one_hot.toarray()                  # only when you actually need it
print(dense[:3])
# [[0. 0. 0. 1. 0.]
#  [0. 0. 0. 1. 0.]
#  [0. 0. 0. 1. 0.]]

Why the output is sparse

A one-hot matrix is one 1 per row and zeros everywhere else. Storing the zeros is waste:

$$\text{dense bytes} = m \times k \times 8 \qquad \text{sparse bytes} \approx m \times (8 + 4) + \text{overhead}$$

Rows	Categories	Dense	Sparse	Saving
20,640	5	826 KB	248 KB	3×
20,640	100	16.5 MB	248 KB	67×
1,000,000	50,000	400 GB	12 MB	33,000×

The last row is why sparse matrices exist. A one-hot encoding of a million user IDs is impossible dense and routine sparse. .toarray() on it will exhaust your memory, which is why scikit-learn makes you ask.

figure One column per distinct value, with no upper bound matplotlib

Below about 50 distinct values one-hot is the obvious choice. Above a few hundred it starts to dominate the feature matrix, and something else is needed.

The dummy variable trap

Five one-hot columns always sum to exactly 1, so any one is perfectly predictable from the other four. The design matrix becomes rank-deficient, and for models solved by matrix inversion the solution is not unique.

$$x_{\text{<1H OCEAN}} + x_{\text{INLAND}} + x_{\text{ISLAND}} + x_{\text{NEAR BAY}} + x_{\text{NEAR OCEAN}} = 1$$

drop="first" removes one column, making the dropped category the reference level that the intercept absorbs.

Model	Does it matter?
Linear/logistic regression with an intercept, no regularisation	Yes — drop one
Ridge, Lasso, Elastic Net	No — the penalty makes the solution unique anyway
Trees, forests, boosting	No — and dropping a column can hurt interpretability
Statistical inference (p-values, coefficient CIs)	Yes, always

scikit-learn’s default is to keep all columns, which is right for regularised and tree-based models — the majority case.

Unseen categories in production

Your encoder learned five categories. Six months later a row arrives with "NEAR LAKE".

unseen.py

import pandas as pd
from sklearn.preprocessing import OneHotEncoder
 
train = pd.DataFrame({"city": ["London", "Paris", "Tokyo"]})
live = pd.DataFrame({"city": ["London", "Berlin"]})       # Berlin is new
 
strict = OneHotEncoder().fit(train)
try:
    strict.transform(live)
except ValueError as exc:
    print("raised:", str(exc)[:60])
# raised: Found unknown categories ['Berlin'] in column 0 during
 
safe = OneHotEncoder(handle_unknown="ignore").fit(train)
print(safe.transform(live).toarray())
# [[1. 0. 0.]
#  [0. 0. 0.]]      <- Berlin becomes all zeros

Both behaviours are defensible. Raising catches a data-quality problem loudly; handle_unknown="ignore" keeps the service running and encodes the unknown as “none of the above”. For anything serving live traffic, use handle_unknown="ignore" and log the occurrence — an all-zero row is a silent degradation otherwise.

Fitting the encoder on train and test together

OneHotEncoder().fit(pd.concat([train, test])) makes the unseen-category problem disappear, and it does so by leaking. The encoder now knows about categories that only exist in the test set, and your evaluation no longer reflects production, where genuinely new categories will appear.

When one-hot is not an option

Approach	What it does	Cost
Frequency encoding	Replace each category with its count or share	Collides: two categories of equal frequency become identical
Target encoding	Replace with the mean target for that category	Leaks badly unless cross-fitted and smoothed
Hashing	Hash into a fixed number of buckets	Collisions, and no way back to the category
Embeddings	Learn a dense vector per category	Needs a neural network and plenty of data
Grouping	Keep the top $k$, bucket the rest as “other”	Simple, effective, loses tail detail

Grouping is the underrated one. Keeping the 20 most common values and folding the remaining 4,980 into other usually captures most of the signal for a fraction of the width.

Target encoding leaks unless you are careful

Encoding a category by its mean target uses the label, so a row’s own label contributes to its own feature. Cross-fit it — compute the encoding on out-of-fold data only — and smooth rare categories toward the global mean. category_encoders.TargetEncoder and scikit-learn’s TargetEncoder both do this; a hand-rolled groupby().mean() does not.

Pitfalls

OrdinalEncoder sorts alphabetically

['<1H OCEAN', 'INLAND', 'ISLAND', 'NEAR BAY', 'NEAR OCEAN'] is alphabetical order, not domain order. If the order matters, state it in the categories argument.

Calling .toarray() reflexively

On five categories it is harmless. On 50,000 it allocates hundreds of gigabytes. Most scikit-learn estimators accept sparse input directly — check before densifying.

Encoding an identifier

Order IDs, transaction hashes and row UUIDs are categorical in dtype and meaningless as features. One-hot encoding them produces one column per row, and a model that memorises the training set perfectly.

Losing the column names

OneHotEncoder returns an array. get_feature_names_out() gives the names back, and you will want them the moment you look at feature importances.

pch.quizTag pch.quizDefaultTitle

1. Why is OrdinalEncoder a poor choice for ocean_proximity with a linear model?

   A It is slower than one-hot encoding B It asserts that the categories are ordered and evenly spaced — so the model treats ISLAND as numerically between INLAND and NEAR BAY C It cannot handle more than four categories D It produces missing values
   pch.quizShowAnswer

   B — It asserts that the categories are ordered and evenly spaced — so the model treats ISLAND as numerically between INLAND and NEAR BAY — A linear model multiplies the integer code by a coefficient, so the arbitrary alphabetical spacing becomes a real quantitative claim about the categories.

2. Why does OneHotEncoder return a sparse matrix by default?

   A Sparse matrices are faster for every operation B One-hot output is almost entirely zeros, and storing only the non-zero entries can be thousands of times smaller C It is required by scikit-learn's API D To prevent the dummy variable trap
   pch.quizShowAnswer

   B — One-hot output is almost entirely zeros, and storing only the non-zero entries can be thousands of times smaller — One million rows by 50,000 categories is 400 GB dense and about 12 MB sparse. The default protects you from an allocation that cannot succeed.

3. For which model does drop='first' genuinely matter?

   A Random forest B Unregularised linear regression with an intercept, where the redundant column makes the design matrix rank-deficient C Ridge regression D It always matters, for every model
   pch.quizShowAnswer

   B — Unregularised linear regression with an intercept, where the redundant column makes the design matrix rank-deficient — The one-hot columns sum to 1, duplicating the intercept. Regularisation makes the solution unique regardless, and trees are unaffected — so scikit-learn keeps every column by default.

4. A category unseen during training arrives in production. What should a serving pipeline do?

   A Refit the encoder on the new data B Use handle_unknown='ignore' so the row encodes as all zeros, and log the event so the silent degradation is visible C Drop the row D Assign it the most common category
   pch.quizShowAnswer

   B — Use handle_unknown='ignore' so the row encodes as all zeros, and log the event so the silent degradation is visible — Raising takes the service down; ignoring keeps it up but encodes an uninformative row. Logging turns a silent failure into a signal that the encoder needs refitting.

🧪 Try It Yourself

Exercise 1 – Ordinal-encode and inspect the ordering

pch.exerciseTag pch.exerciseTitle

Exercise 2 – One-hot encode and read the names

pch.exerciseTag pch.exerciseTitle

Exercise 3 – Measure the memory saving

pch.exerciseTag pch.exerciseTitle

Exercise 4 – Survive an unseen category

pch.exerciseTag pch.exerciseTitle

Exercise 5 – Confirm the dummy variable trap

pch.exerciseTag pch.exerciseTitle

Recap

• The encoding is a claim about the categories. Ordinal claims an order and a spacing; one-hot claims neither.
• OrdinalEncoder sorts alphabetically — pass categories=[[...]] whenever the order is real.
• Trees can live with ordinal encoding because they split on thresholds; linear models cannot.
• One-hot output is sparse because it must be: a million rows by 50,000 categories is 400 GB dense and 12 MB sparse.
• The one-hot columns sum to 1, so with an unregularised intercept model drop one. Regularised and tree-based models do not care.
• Use handle_unknown="ignore" in production, and log every occurrence.
• Above a few hundred categories, group into a top-$k$ plus “other”, or use target encoding done properly with cross-fitting.

Exercise 6 – Measure what each encoding claims

pch.exerciseTag pch.exerciseTitle

Next

Continue to Feature Scaling (Normalization & Standardization) — every column is numeric now, and they span four orders of magnitude.