Skip to content
Python Central Hub

Text Preprocessing (Tokenization, Stemming, Lemmatization)

In plain words

Before any neural network can “read” text, the text has to become numbers. That’s the whole job of preprocessing: chop a sentence into pieces, clean those pieces up, and give each one an ID. Everything downstream — embeddings, RNNs, attention, Transformers — starts from this step.

Why preprocessing exists

Text is messy:

  • punctuation
  • capitalization
  • emojis
  • spelling variations

Preprocessing reduces noise and makes feature extraction more consistent.

Tokenization

Tokenization splits text into units (tokens):

  • words
  • subwords
  • characters

Example:

“I love NLP!” → [“I”, “love”, “NLP”]

Stemming

Stemming cuts words down to a root form:

  • “running” → “run”
  • “studies” → “studi” (can be rough)

Pros:

  • simple, fast

Cons:

  • can create non-words

Lemmatization

Lemmatization converts to the dictionary form (lemma):

  • “better” → “good”
  • “running” → “run”

Pros:

  • more accurate than stemming

Cons:

  • slower, needs language rules

Common steps

  • lowercasing
  • removing extra spaces
  • removing/keeping punctuation depending on task
  • stop word removal (sometimes)
diagram Diagram mermaid

Two levels of tokenization (from the book)

Hands-On Machine Learning builds its Char-RNN using Keras’s TokenizerTokenizer class with char_level=Truechar_level=True, so every character — not word — gets its own ID:

Character-level vs word-level tokenizers
import tensorflow as tf
 
text = "First we tokenize"
 
# Character-level tokenizer (used for the book's Shakespeare Char-RNN)
char_tokenizer = tf.keras.preprocessing.text.Tokenizer(char_level=True)
char_tokenizer.fit_on_texts([text])
print("Char IDs:", char_tokenizer.texts_to_sequences(["First"]))
print("Distinct characters:", len(char_tokenizer.word_index))
 
# Word-level tokenizer (the default, used for IMDb sentiment analysis)
word_tokenizer = tf.keras.preprocessing.text.Tokenizer(char_level=False)
word_tokenizer.fit_on_texts([text])
print("Word IDs:", word_tokenizer.texts_to_sequences(["First we tokenize"]))
print("Distinct words:", len(word_tokenizer.word_index))
Character-level vs word-level tokenizers
import tensorflow as tf
 
text = "First we tokenize"
 
# Character-level tokenizer (used for the book's Shakespeare Char-RNN)
char_tokenizer = tf.keras.preprocessing.text.Tokenizer(char_level=True)
char_tokenizer.fit_on_texts([text])
print("Char IDs:", char_tokenizer.texts_to_sequences(["First"]))
print("Distinct characters:", len(char_tokenizer.word_index))
 
# Word-level tokenizer (the default, used for IMDb sentiment analysis)
word_tokenizer = tf.keras.preprocessing.text.Tokenizer(char_level=False)
word_tokenizer.fit_on_texts([text])
print("Word IDs:", word_tokenizer.texts_to_sequences(["First we tokenize"]))
print("Distinct words:", len(word_tokenizer.word_index))
text
Char IDs: [[6, 3, 1, 7, 2]]
Distinct characters: 9
Word IDs: [[3, 4, 1]]
Distinct words: 4
text
Char IDs: [[6, 3, 1, 7, 2]]
Distinct characters: 9
Word IDs: [[3, 4, 1]]
Distinct words: 4

The three steps every vectorizer follows

Whichever library you reach for, turning text into numbers always follows the same three-step template:

  1. Standardize — lowercase it, strip punctuation, so “Sunset” and “sunset” don’t end up as two different tokens.
  2. Tokenize — split the standardized text into units (words, subwords, or characters).
  3. Index — assign each distinct token an integer ID, using a vocabulary built from the training data.

Keras bundles all three steps into a single layer: TextVectorizationTextVectorization. It standardizes, tokenizes, and indexes text in one call, and — because it’s a real Keras layer — it can live directly inside a tf.datatf.data pipeline or inside the model itself.

The Keras TextVectorization layer
from tensorflow.keras.layers import TextVectorization
 
text_vectorization = TextVectorization(output_mode="int")
 
dataset = [
    "I write, erase, rewrite",
    "Erase again, and then",
    "A poppy blooms.",
]
text_vectorization.adapt(dataset)   # builds the vocabulary from the dataset
 
test_sentence = "I write, rewrite, and still rewrite again"
encoded_sentence = text_vectorization(test_sentence)
print("Encoded:", encoded_sentence.numpy())
 
vocabulary = text_vectorization.get_vocabulary()
inverse_vocab = dict(enumerate(vocabulary))
decoded_sentence = " ".join(inverse_vocab[int(i)] for i in encoded_sentence)
print("Decoded:", decoded_sentence)
The Keras TextVectorization layer
from tensorflow.keras.layers import TextVectorization
 
text_vectorization = TextVectorization(output_mode="int")
 
dataset = [
    "I write, erase, rewrite",
    "Erase again, and then",
    "A poppy blooms.",
]
text_vectorization.adapt(dataset)   # builds the vocabulary from the dataset
 
test_sentence = "I write, rewrite, and still rewrite again"
encoded_sentence = text_vectorization(test_sentence)
print("Encoded:", encoded_sentence.numpy())
 
vocabulary = text_vectorization.get_vocabulary()
inverse_vocab = dict(enumerate(vocabulary))
decoded_sentence = " ".join(inverse_vocab[int(i)] for i in encoded_sentence)
print("Decoded:", decoded_sentence)
text
Encoded: [ 7  3  5  9  1  5 10]
Decoded: i write rewrite and [UNK] rewrite again
text
Encoded: [ 7  3  5  9  1  5 10]
Decoded: i write rewrite and [UNK] rewrite again

Two vocabulary slots are always reserved before any real word gets an ID:

  • index 0 — the mask token, used to pad shorter sequences so a batch can be one contiguous tensor; it means “ignore me, I’m not a real word.”
  • index 1 — the [UNK][UNK] (out-of-vocabulary) token, a catch-all for any word the vectorizer never saw during adapt()adapt() — like "rewrite""rewrite"’s cousin "still""still" above, which got mapped to [UNK][UNK] because it wasn’t common enough to make the vocabulary.

Visualize it

Here’s how raw text moves through the preprocessing pipeline before it reaches a model.

diagram NLP preprocessing pipeline mermaid
How raw text is tokenized, cleaned, and reduced to root forms before modeling.

Mini-checkpoint

Should you always remove stop words?

  • Not always. For sentiment tasks, words like “not” are critical.

🧪 Try It Yourself

Exercise 1 – Tokenize a Sentence

Exercise 2 – Strip Punctuation Before Tokenizing

Exercise 3 – A Tiny Stemmer

Next

Continue to Bag of Words (BoW) & TF-IDF — turn these clean tokens into the numeric vectors a model can actually train on.

If this helped you, consider buying me a coffee ☕

Buy me a coffee

Was this page helpful?

Let us know how we did