What you’ll learn

• why OrdinalEncoderOrdinalEncoder makes a claim about your data, and when that claim is false
• one-hot encoding, and the geometric reason it is the safe default
• the memory arithmetic behind sparse matrices — 20,640 × 5 stored in 165 KB or 826 KB
• the dummy variable trap, and when drop="first"drop="first" matters and when it does not
• what happens when an unseen category arrives in production
• four alternatives for high-cardinality columns, with the cost of each

Intuition

Almost no model accepts the string "NEAR BAY""NEAR BAY". Encoding turns text into numbers, and the encoding you pick is not a formatting decision — it tells the model what it is allowed to believe about those categories.

Map five categories to 0–4 and you have asserted that they lie on a line, evenly spaced, in that order. Sometimes true (small < medium < largesmall < medium < large). Here, false: ISLANDISLAND is not “between” INLANDINLAND and NEAR BAYNEAR BAY in any sense, yet a linear model will treat the numbers exactly that way.

  flowchart LR
  T["Text column"] --> Q{"Do the categories
have a real order?"}
  Q -->|yes| O["OrdinalEncoder
1 column, ordering preserved"]
  Q -->|no| C{"How many
distinct values?"}
  C -->|"under ~50"| H["OneHotEncoder
one column each"]
  C -->|"hundreds or more"| A["Target / frequency /
hashing / embedding"]

The one text column

print(housing["ocean_proximity"].value_counts())
 
# <1H OCEAN     9136
# INLAND        6551
# NEAR OCEAN    2658
# NEAR BAY      2290
# ISLAND           5

print(housing["ocean_proximity"].value_counts())
 
# <1H OCEAN     9136
# INLAND        6551
# NEAR OCEAN    2658
# NEAR BAY      2290
# ISLAND           5

figureFive categories spanning three orders of magnitudematplotlib

Ordinal encoding

from sklearn.preprocessing import OrdinalEncoder
 
encoder = OrdinalEncoder()
encoded = encoder.fit_transform(housing[["ocean_proximity"]])
 
print(encoder.categories_)
# [array(['<1H OCEAN', 'INLAND', 'ISLAND', 'NEAR BAY', 'NEAR OCEAN'], dtype=object)]
print(encoded[:5].ravel())      # [3. 3. 3. 3. 3.]

from sklearn.preprocessing import OrdinalEncoder
 
encoder = OrdinalEncoder()
encoded = encoder.fit_transform(housing[["ocean_proximity"]])
 
print(encoder.categories_)
# [array(['<1H OCEAN', 'INLAND', 'ISLAND', 'NEAR BAY', 'NEAR OCEAN'], dtype=object)]
print(encoded[:5].ravel())      # [3. 3. 3. 3. 3.]

Categories are assigned integers in alphabetical order, which is where the trouble starts. ISLANDISLAND becomes 2 purely because “I” sorts between “I” and “N” — and a linear model then reads that as: an island district is numerically between an inland one and a bay one, and twice as far from <1H OCEAN<1H OCEAN as INLANDINLAND is.

figureWhat each encoding assertsmatplotlib

Geometrically: under one-hot every pair of categories is $\sqrt{2}$ apart, so no ordering and no spacing is implied. Under ordinal the distances are 1, 2, 3 and 4, and the model uses them.

See it move

Distances are the whole argument, so the sketch computes them. Hover a category to select it; the bars show how far every other category sits from your selection under each encoding.

The amber bars are the encoder’s uninvited claim. Nothing in the data says ISLANDISLAND sits between INLANDINLAND and NEAR BAYNEAR BAY; alphabetical order put it there, and a linear model reads it as a distance. The green bars assert only “different”, which is all that is true.

Tree models are the exception. A tree splits on thresholds, so ordinal <= 1.5ordinal <= 1.5 can isolate any group of categories the ordering happens to make contiguous. Ordinal encoding for trees is often fine and much cheaper than one-hot, which is why gradient boosting libraries offer native categorical support built on exactly this idea.

One-hot encoding

from sklearn.preprocessing import OneHotEncoder
 
encoder = OneHotEncoder()
one_hot = encoder.fit_transform(housing[["ocean_proximity"]])
 
print(one_hot)
# <Compressed Sparse Row sparse matrix of dtype 'float64'
#   with 20640 stored elements and shape (20640, 5)>
 
print(one_hot.shape)                       # (20640, 5)
print(encoder.get_feature_names_out())
# ['ocean_proximity_<1H OCEAN' ... 'ocean_proximity_NEAR OCEAN']
 
dense = one_hot.toarray()                  # only when you actually need it
print(dense[:3])
# [[0. 0. 0. 1. 0.]
#  [0. 0. 0. 1. 0.]
#  [0. 0. 0. 1. 0.]]

from sklearn.preprocessing import OneHotEncoder
 
encoder = OneHotEncoder()
one_hot = encoder.fit_transform(housing[["ocean_proximity"]])
 
print(one_hot)
# <Compressed Sparse Row sparse matrix of dtype 'float64'
#   with 20640 stored elements and shape (20640, 5)>
 
print(one_hot.shape)                       # (20640, 5)
print(encoder.get_feature_names_out())
# ['ocean_proximity_<1H OCEAN' ... 'ocean_proximity_NEAR OCEAN']
 
dense = one_hot.toarray()                  # only when you actually need it
print(dense[:3])
# [[0. 0. 0. 1. 0.]
#  [0. 0. 0. 1. 0.]
#  [0. 0. 0. 1. 0.]]

Why the output is sparse

A one-hot matrix is one 1 per row and zeros everywhere else. Storing the zeros is waste:

The last row is why sparse matrices exist. A one-hot encoding of a million user IDs is impossible dense and routine sparse. .toarray().toarray() on it will exhaust your memory, which is why scikit-learn makes you ask.

figureOne column per distinct value, with no upper boundmatplotlib

The dummy variable trap

Five one-hot columns always sum to exactly 1, so any one is perfectly predictable from the other four. The design matrix becomes rank-deficient, and for models solved by matrix inversion the solution is not unique.

drop="first"drop="first" removes one column, making the dropped category the reference level that the intercept absorbs.

scikit-learn’s default is to keep all columns, which is right for regularised and tree-based models — the majority case.

Unseen categories in production

Your encoder learned five categories. Six months later a row arrives with "NEAR LAKE""NEAR LAKE".

import pandas as pd
from sklearn.preprocessing import OneHotEncoder
 
train = pd.DataFrame({"city": ["London", "Paris", "Tokyo"]})
live = pd.DataFrame({"city": ["London", "Berlin"]})       # Berlin is new
 
strict = OneHotEncoder().fit(train)
try:
    strict.transform(live)
except ValueError as exc:
    print("raised:", str(exc)[:60])
# raised: Found unknown categories ['Berlin'] in column 0 during
 
safe = OneHotEncoder(handle_unknown="ignore").fit(train)
print(safe.transform(live).toarray())
# [[1. 0. 0.]
#  [0. 0. 0.]]      <- Berlin becomes all zeros

import pandas as pd
from sklearn.preprocessing import OneHotEncoder
 
train = pd.DataFrame({"city": ["London", "Paris", "Tokyo"]})
live = pd.DataFrame({"city": ["London", "Berlin"]})       # Berlin is new
 
strict = OneHotEncoder().fit(train)
try:
    strict.transform(live)
except ValueError as exc:
    print("raised:", str(exc)[:60])
# raised: Found unknown categories ['Berlin'] in column 0 during
 
safe = OneHotEncoder(handle_unknown="ignore").fit(train)
print(safe.transform(live).toarray())
# [[1. 0. 0.]
#  [0. 0. 0.]]      <- Berlin becomes all zeros

Both behaviours are defensible. Raising catches a data-quality problem loudly; handle_unknown="ignore"handle_unknown="ignore" keeps the service running and encodes the unknown as “none of the above”. For anything serving live traffic, use handle_unknown="ignore"handle_unknown="ignore" and log the occurrence — an all-zero row is a silent degradation otherwise.

When one-hot is not an option

Grouping is the underrated one. Keeping the 20 most common values and folding the remaining 4,980 into otherother usually captures most of the signal for a fraction of the width.

Pitfalls

🧪 Try It Yourself

Exercise 1 – Ordinal-encode and inspect the ordering

Exercise 2 – One-hot encode and read the names

Exercise 3 – Measure the memory saving

Exercise 4 – Survive an unseen category

Exercise 5 – Confirm the dummy variable trap

Recap

• The encoding is a claim about the categories. Ordinal claims an order and a spacing; one-hot claims neither.
• OrdinalEncoderOrdinalEncoder sorts alphabetically — pass categories=[[...]]categories=[[...]] whenever the order is real.
• Trees can live with ordinal encoding because they split on thresholds; linear models cannot.
• One-hot output is sparse because it must be: a million rows by 50,000 categories is 400 GB dense and 12 MB sparse.
• The one-hot columns sum to 1, so with an unregularised intercept model drop one. Regularised and tree-based models do not care.
• Use handle_unknown="ignore"handle_unknown="ignore" in production, and log every occurrence.
• Above a few hundred categories, group into a top-$k$ plus “other”, or use target encoding done properly with cross-fitting.

Exercise 6 – Measure what each encoding claims

Next

Continue to Feature Scaling (Normalization & Standardization) — every column is numeric now, and they span four orders of magnitude.