Skip to content
Python Central Hub

Categorical Data Type

What you’ll learn

  • Why repeated string values (like "apple""apple", "apple""apple", "orange""orange"…) waste memory
  • How pandas’ categorycategory dtype stores data as categories + integer codes
  • Converting a column with astype("category")astype("category"), and building one with pd.Categoricalpd.Categorical
  • Ordered categoricals and the .cat.cat accessor (categoriescategories, codescodes, rename_categoriesrename_categories, set_categoriesset_categories, as_orderedas_ordered, remove_unused_categoriesremove_unused_categories)
  • Why groupbygroupby and value_countsvalue_counts are faster on categoricals
  • A quick look at nullable extension dtypes: Int64Int64, stringstring, booleanboolean, and pd.NApd.NA

Why a “category” dtype?

Real datasets are full of columns that only hold a handful of distinct values, repeated over and over. A fruitfruit column with a million rows might only ever say "apple""apple" or "orange""orange". Storing every one of those million rows as a full Python string is wasteful — you’re storing the same few words again and again.

pandas solves this the way data warehouses have for decades: keep a small table of the distinct values (the categories), and store each row as a cheap integer code that points into that table. This is called the categorical (or dictionary-encoded) representation.

the idea, by hand
import pandas as pd
 
values = pd.Series([0, 1, 0, 0] * 2)      # integer codes
dim = pd.Series(["apple", "orange"])      # the categories (dimension table)
 
print(dim.take(values).tolist())
the idea, by hand
import pandas as pd
 
values = pd.Series([0, 1, 0, 0] * 2)      # integer codes
dim = pd.Series(["apple", "orange"])      # the categories (dimension table)
 
print(dim.take(values).tolist())
text
# Expected output:
# ['apple', 'orange', 'apple', 'apple', 'apple', 'orange', 'apple', 'apple']
text
# Expected output:
# ['apple', 'orange', 'apple', 'apple', 'apple', 'orange', 'apple', 'apple']

dim.take(values)dim.take(values) rebuilds the original strings from the tiny lookup table — that’s exactly what a CategoricalCategorical does internally, automatically.

Converting a column with astype("category")astype("category")

fruit_categorical.py
import numpy as np
import pandas as pd
 
fruits = ["apple", "orange", "apple", "apple"] * 2
df = pd.DataFrame({
    "fruit": fruits,
    "basket_id": np.arange(len(fruits)),
})
 
fruit_cat = df["fruit"].astype("category")
print(fruit_cat)
fruit_categorical.py
import numpy as np
import pandas as pd
 
fruits = ["apple", "orange", "apple", "apple"] * 2
df = pd.DataFrame({
    "fruit": fruits,
    "basket_id": np.arange(len(fruits)),
})
 
fruit_cat = df["fruit"].astype("category")
print(fruit_cat)
text
# Expected output:
# 0     apple
# 1    orange
# 2     apple
# 3     apple
# 4     apple
# 5    orange
# 6     apple
# 7     apple
# Name: fruit, dtype: category
# Categories (2, str): ['apple', 'orange']
text
# Expected output:
# 0     apple
# 1    orange
# 2     apple
# 3     apple
# 4     apple
# 5    orange
# 6     apple
# 7     apple
# Name: fruit, dtype: category
# Categories (2, str): ['apple', 'orange']

Under the hood, fruit_catfruit_cat now holds a pandas.Categoricalpandas.Categorical. You can pull the two pieces apart with the .cat.cat accessor:

categories_and_codes.py
print(fruit_cat.cat.categories.tolist())
print(fruit_cat.cat.codes.tolist())
categories_and_codes.py
print(fruit_cat.cat.categories.tolist())
print(fruit_cat.cat.codes.tolist())
text
# Expected output:
# ['apple', 'orange']
# [0, 1, 0, 0, 0, 1, 0, 0]
text
# Expected output:
# ['apple', 'orange']
# [0, 1, 0, 0, 0, 1, 0, 0]

categoriescategories is the small lookup table. codescodes is the compact integer array that replaces the repeated strings — one small int per row instead of one full string per row.

Building a Categorical directly

You don’t have to start from a plain Series. pd.Categoricalpd.Categorical and pd.Categorical.from_codespd.Categorical.from_codes build one directly:

build_categorical.py
import pandas as pd
 
my_categories = pd.Categorical(["foo", "bar", "baz", "foo", "bar"])
print(my_categories)
 
categories = ["foo", "bar", "baz"]
codes = [0, 1, 2, 0, 0, 1]
from_codes = pd.Categorical.from_codes(codes, categories)
print(from_codes)
build_categorical.py
import pandas as pd
 
my_categories = pd.Categorical(["foo", "bar", "baz", "foo", "bar"])
print(my_categories)
 
categories = ["foo", "bar", "baz"]
codes = [0, 1, 2, 0, 0, 1]
from_codes = pd.Categorical.from_codes(codes, categories)
print(from_codes)
text
# Expected output:
# ['foo', 'bar', 'baz', 'foo', 'bar']
# Categories (3, str): ['bar', 'baz', 'foo']
# ['foo', 'bar', 'baz', 'foo', 'foo', 'bar']
# Categories (3, str): ['foo', 'bar', 'baz']
text
# Expected output:
# ['foo', 'bar', 'baz', 'foo', 'bar']
# Categories (3, str): ['bar', 'baz', 'foo']
# ['foo', 'bar', 'baz', 'foo', 'foo', 'bar']
# Categories (3, str): ['foo', 'bar', 'baz']

Notice the categories print in a different order than you passed them in — by default pandas just sorts the distinct values it finds. If the categories have a real ranking (like survey answers “low” < “medium” < “high”), pass ordered=Trueordered=True so comparisons like << work correctly:

ordered_categorical.py
ordered_cat = pd.Categorical.from_codes(codes, categories, ordered=True)
print(ordered_cat)
ordered_categorical.py
ordered_cat = pd.Categorical.from_codes(codes, categories, ordered=True)
print(ordered_cat)
text
# Expected output:
# ['foo', 'bar', 'baz', 'foo', 'foo', 'bar']
# Categories (3, str): ['foo' < 'bar' < 'baz']
text
# Expected output:
# ['foo', 'bar', 'baz', 'foo', 'foo', 'bar']
# Categories (3, str): ['foo' < 'bar' < 'baz']

An existing categorical can be flipped to ordered later with .cat.as_ordered().cat.as_ordered().

Editing categories with the .cat.cat accessor

The categories can be renamed, expanded, or trimmed without touching the codes, which makes these operations cheap:

cat_accessor_methods.py
s = pd.Series(["a", "b", "c", "d"] * 2).astype("category")
 
# Add a category that doesn't appear in the data yet
s2 = s.cat.set_categories(["a", "b", "c", "d", "e"])
print(s2.value_counts().sort_index())
 
# Rename categories in place (order-preserving)
s3 = s.cat.rename_categories(["A", "B", "C", "D"])
print(s3.cat.categories.tolist())
 
# Drop categories that no longer appear after filtering
s4 = s[s.isin(["a", "b"])]
print(s4.cat.categories.tolist())                       # still has c, d
print(s4.cat.remove_unused_categories().cat.categories.tolist())
cat_accessor_methods.py
s = pd.Series(["a", "b", "c", "d"] * 2).astype("category")
 
# Add a category that doesn't appear in the data yet
s2 = s.cat.set_categories(["a", "b", "c", "d", "e"])
print(s2.value_counts().sort_index())
 
# Rename categories in place (order-preserving)
s3 = s.cat.rename_categories(["A", "B", "C", "D"])
print(s3.cat.categories.tolist())
 
# Drop categories that no longer appear after filtering
s4 = s[s.isin(["a", "b"])]
print(s4.cat.categories.tolist())                       # still has c, d
print(s4.cat.remove_unused_categories().cat.categories.tolist())
text
# Expected output:
# a    2
# b    2
# c    2
# d    2
# e    0
# Name: count, dtype: int64
# ['A', 'B', 'C', 'D']
# ['a', 'b', 'c', 'd']
# ['a', 'b']
text
# Expected output:
# a    2
# b    2
# c    2
# d    2
# e    0
# Name: count, dtype: int64
# ['A', 'B', 'C', 'D']
# ['a', 'b', 'c', 'd']
# ['a', 'b']

Computing with categoricals

groupbygroupby and value_countsvalue_counts treat a categorical column the same way they’d treat a plain string column — but faster, because they work on the small integer codes array instead of comparing full strings.

groupby_categorical.py
import numpy as np
import pandas as pd
 
rng = np.random.default_rng(seed=12345)
draws = rng.standard_normal(1000)
 
bins = pd.qcut(draws, 4, labels=["Q1", "Q2", "Q3", "Q4"])
bins = pd.Series(bins, name="quartile")
 
results = (
    pd.Series(draws)
    .groupby(bins, observed=False)
    .agg(["count", "min", "max"])
    .reset_index()
)
print(results["quartile"].dtype)
groupby_categorical.py
import numpy as np
import pandas as pd
 
rng = np.random.default_rng(seed=12345)
draws = rng.standard_normal(1000)
 
bins = pd.qcut(draws, 4, labels=["Q1", "Q2", "Q3", "Q4"])
bins = pd.Series(bins, name="quartile")
 
results = (
    pd.Series(draws)
    .groupby(bins, observed=False)
    .agg(["count", "min", "max"])
    .reset_index()
)
print(results["quartile"].dtype)
text
# Expected output:
# category
text
# Expected output:
# category

pd.qcutpd.qcut and pd.cutpd.cut already hand you back a CategoricalCategorical — that’s why binned columns are cheap to group by, even across millions of rows.

Diagram: how a categorical is stored

diagram String column vs categorical representation mermaid
A repeated string column collapses into a small categories table plus a compact integer codes array.

See the memory shrink

sketch Repeated strings collapsing into codes + a lookup table p5.js
Full-length strings on the left compress into a tiny integer codes array plus a small categories table on the right.

A note on nullable extension dtypes

categorycategory is one of several extension dtypes pandas added on top of the original NumPy-based system. NumPy has no clean way to mark an integer or boolean as missing, so plain pandas used to silently upcast a column of integers with a missing value to float64float64 (using NaNNaN). Extension dtypes fix that by using a dedicated missing marker, pd.NApd.NA, instead:

nullable_dtypes.py
import pandas as pd
 
s = pd.Series([1, 2, 3, None], dtype="Int64")
print(s)
print(s.isna().tolist())
nullable_dtypes.py
import pandas as pd
 
s = pd.Series([1, 2, 3, None], dtype="Int64")
print(s)
print(s.isna().tolist())
text
# Expected output:
# 0       1
# 1       2
# 2       3
# 3    <NA>
# dtype: Int64
# [False, False, False, True]
text
# Expected output:
# 0       1
# 1       2
# 2       3
# 3    <NA>
# dtype: Int64
# [False, False, False, True]

The capitalized "Int64""Int64" (as opposed to lowercase "int64""int64") is what tells pandas to use the nullable extension type. The same pattern gives you "string""string" for text and "boolean""boolean" for True/False columns that may contain nulls — handy when you’re cleaning a dataset that mixes real values with missing ones.

🧪 Try It Yourself

Exercise 1 – Convert a column to category dtype

Exercise 2 – Read the integer codes

Exercise 3 – Prove the memory savings

Next

Now that repeated string columns can be compact and fast, move on to Introduction to Matplotlib to start visualizing the data you’ve cleaned and encoded.

If this helped you, consider buying me a coffee ☕

Buy me a coffee

Was this page helpful?

Let us know how we did