In plain words

Once text is tokenized, a model still can’t use it directly — a word like "python""python" isn’t a number. Bag of Words and TF-IDF are the two simplest ways to turn a pile of tokens into a vector: BoW just counts words, while TF-IDF counts them and downweights the boring, everywhere words so the interesting ones stand out.

Bag of Words (BoW)

BoW represents a document by:

• counting how many times each word appears

Key idea:

• word order is ignored

  flowchart LR
  D[Document] --> V[Vector of word counts]

Pros:

• simple and fast
• strong baseline

Cons:

• ignores word order and context
• common words dominate

TF-IDF

TF-IDF reduces the impact of very common words by weighting them lower.

Intuition:

• words common in this doc but rare across all docs are informative

  flowchart LR
  TF[Term Frequency] --> W[Weight]
  IDF[Inverse Document Frequency] --> W
  W --> V[TF-IDF vector]

Scikit-learn examples

from sklearn.feature_extraction.text import CountVectorizer
 
vec = CountVectorizer()
X = vec.fit_transform(["i love python", "python is great"])

from sklearn.feature_extraction.text import CountVectorizer
 
vec = CountVectorizer()
X = vec.fit_transform(["i love python", "python is great"])

from sklearn.feature_extraction.text import TfidfVectorizer
 
docs = ["i love python", "python is great", "i love great libraries"]
vec = TfidfVectorizer()
X = vec.fit_transform(docs)
 
print("Vocabulary:", vec.get_feature_names_out())
print("TF-IDF matrix:\n", X.toarray().round(2))

from sklearn.feature_extraction.text import TfidfVectorizer
 
docs = ["i love python", "python is great", "i love great libraries"]
vec = TfidfVectorizer()
X = vec.fit_transform(docs)
 
print("Vocabulary:", vec.get_feature_names_out())
print("TF-IDF matrix:\n", X.toarray().round(2))

Vocabulary: ['great' 'is' 'libraries' 'love' 'python']
TF-IDF matrix:
 [[0.   0.   0.   0.71 0.71]
 [0.52 0.68 0.   0.   0.52]
 [0.52 0.   0.68 0.52 0.  ]]

Vocabulary: ['great' 'is' 'libraries' 'love' 'python']
TF-IDF matrix:
 [[0.   0.   0.   0.71 0.71]
 [0.52 0.68 0.   0.   0.52]
 [0.52 0.   0.68 0.52 0.  ]]

Notice "great""great" gets a lower weight than "libraries""libraries" in the third document — "great""great" shows up in two of the three documents (less discriminative), while "libraries""libraries" only appears once (more discriminative).

Visualize it

TF-IDF gives every term in a document its own weight. Click a document to see how its term weights compare — common words across the corpus (like “python” here) get pushed down, while rarer, document-specific words get pushed up.

The same idea with Keras’s TextVectorization layer

CountVectorizerCountVectorizer and TfidfVectorizerTfidfVectorizer are scikit-learn’s tools for this, but Keras’s TextVectorizationTextVectorization layer can produce the exact same kind of vectors directly inside a model, just by changing its output_modeoutput_mode:

from tensorflow.keras.layers import TextVectorization
 
docs = ["i love python", "python is great", "i love great libraries"]
 
# multi_hot: a 0/1 flag per word -- did this word appear at all?
binary_vectorizer = TextVectorization(max_tokens=20, output_mode="multi_hot")
binary_vectorizer.adapt(docs)
 
# tf_idf: built-in term-frequency / inverse-document-frequency weighting
tfidf_vectorizer = TextVectorization(max_tokens=20, output_mode="tf_idf")
tfidf_vectorizer.adapt(docs)
 
print("Vocabulary:", binary_vectorizer.get_vocabulary())

from tensorflow.keras.layers import TextVectorization
 
docs = ["i love python", "python is great", "i love great libraries"]
 
# multi_hot: a 0/1 flag per word -- did this word appear at all?
binary_vectorizer = TextVectorization(max_tokens=20, output_mode="multi_hot")
binary_vectorizer.adapt(docs)
 
# tf_idf: built-in term-frequency / inverse-document-frequency weighting
tfidf_vectorizer = TextVectorization(max_tokens=20, output_mode="tf_idf")
tfidf_vectorizer.adapt(docs)
 
print("Vocabulary:", binary_vectorizer.get_vocabulary())

Pass ngrams=2ngrams=2 to either vectorizer and it will index bigrams instead of single words — the exact same “reinject a bit of local word order” trick that CountVectorizer(ngram_range=(1, 2))CountVectorizer(ngram_range=(1, 2)) performs above.

Bag-of-words or sequence model? A rule of thumb

Bag-of-words throws away word order entirely, which raises an obvious question: when is that actually fine, and when do you need a model that reads words in order (an RNN or a Transformer)? Chollet’s team ran a systematic study across many text-classification datasets and found a surprisingly reliable heuristic — look at the ratio between your number of training samples and the average number of words per sample:

The intuition: a bag-of-words feature space is simple, so a small dataset is enough to fit it well. A sequence model has to learn a much richer space — which order and how much text matters in — so it only pays off once you have plenty of data to fill that space with.

Mini-checkpoint

Why is TF-IDF often better than raw counts?

• it downweights words that aren’t discriminative across the corpus.

🧪 Try It Yourself

Exercise 1 – Build a Bag of Words

Exercise 2 – Weight Terms with TF-IDF

Exercise 3 – Count Words Like the Book Does

Next

Continue to Word Embeddings (Word2Vec, GloVe) — instead of sparse counts, give every word a small, dense vector that captures meaning.