In plain words

BoW and TF-IDF give every word its own slot in a giant, mostly-empty vector — they have no idea that “great” and “fantastic” mean almost the same thing. Embeddings fix that: every word becomes a short, dense vector, learned so that words used in similar contexts end up with similar vectors.

What embeddings are

Embeddings map words to dense vectors:

• similar words have similar vectors

Example intuition:

• king and queen vectors are close
• dog and puppy vectors are close

Why embeddings are better than BoW (sometimes)

BoW/TF-IDF are sparse and don’t capture similarity.

Embeddings capture:

• semantic relationships
• analogies (to some extent)

  flowchart LR
  W[Word] --> E["Embedding vector (dense)"]
  E --> M[Model]

Word2Vec

Word2Vec doesn’t need labeled data at all — the sentence itself provides the “labels.” It comes in two flavors, and both are trained the same way a classifier would be, just on a task invented purely from raw text:

• CBOW (Continuous Bag of Words) predicts a missing word from its surrounding context: given ["the", ___, "sat", "down"]["the", ___, "sat", "down"], predict "cat""cat".
• Skip-gram runs the same idea backwards: given "cat""cat", predict the words likely to appear near it ("the""the", "sat""sat", "down""down").

Either direction pushes two words toward similar vectors whenever they show up in similar contexts — which is exactly why “cat” and “kitten” end up close together despite never being told they’re related.

GloVe

Word2Vec learns by sliding a small window over text and looking at local context. GloVe (Global Vectors) instead starts from a big table of global word co-occurrence counts — how often every pair of words appears together across the entire corpus — and fits vectors so that the dot product between two word vectors approximates the log of how often those words co-occur. In practice, both approaches converge on embedding spaces with very similar properties (synonym clustering, the analogy directions below), and both ship as free, pretrained downloads for English.

Using pretrained vectors instead of training your own

Training good embeddings from scratch needs a lot of text. When your dataset is small, it’s usually better to download vectors someone already trained on billions of words and copy them straight into your model. The pattern is always the same three steps: parse the pretrained file into a {word: vector}{word: vector} lookup, build a matrix indexed by your vocabulary, then hand that matrix to an EmbeddingEmbedding layer as its starting weights.

import numpy as np
 
# a couple of toy vectors, standing in for lines parsed from glove.6B.100d.txt
glove_vectors = {
    "cat": np.array([0.23, -0.15, 0.88]),
    "dog": np.array([0.21, -0.11, 0.91]),
    "car": np.array([-0.60, 0.44, 0.02]),
}
 
vocabulary = {"cat": 1, "dog": 2, "car": 3}   # word -> row index (0 stays reserved for padding)
embedding_dim = 3
embedding_matrix = np.zeros((len(vocabulary) + 1, embedding_dim))
 
for word, i in vocabulary.items():
    vector = glove_vectors.get(word)
    if vector is not None:
        embedding_matrix[i] = vector
 
print(embedding_matrix)

import numpy as np
 
# a couple of toy vectors, standing in for lines parsed from glove.6B.100d.txt
glove_vectors = {
    "cat": np.array([0.23, -0.15, 0.88]),
    "dog": np.array([0.21, -0.11, 0.91]),
    "car": np.array([-0.60, 0.44, 0.02]),
}
 
vocabulary = {"cat": 1, "dog": 2, "car": 3}   # word -> row index (0 stays reserved for padding)
embedding_dim = 3
embedding_matrix = np.zeros((len(vocabulary) + 1, embedding_dim))
 
for word, i in vocabulary.items():
    vector = glove_vectors.get(word)
    if vector is not None:
        embedding_matrix[i] = vector
 
print(embedding_matrix)

[[ 0.    0.    0.  ]
 [ 0.23 -0.15  0.88]
 [ 0.21 -0.11  0.91]
 [-0.6   0.44  0.02]]

[[ 0.    0.    0.  ]
 [ 0.23 -0.15  0.88]
 [ 0.21 -0.11  0.91]
 [-0.6   0.44  0.02]]

Then plug that matrix straight into an EmbeddingEmbedding layer as its starting weights, and freeze it so training doesn’t overwrite the pretrained vectors:

import tensorflow as tf
 
embedding_layer = tf.keras.layers.Embedding(
    input_dim=embedding_matrix.shape[0],
    output_dim=embedding_matrix.shape[1],
    embeddings_initializer=tf.keras.initializers.Constant(embedding_matrix),
    trainable=False,   # keep GloVe's vectors frozen instead of fine-tuning them
)

import tensorflow as tf
 
embedding_layer = tf.keras.layers.Embedding(
    input_dim=embedding_matrix.shape[0],
    output_dim=embedding_matrix.shape[1],
    embeddings_initializer=tf.keras.initializers.Constant(embedding_matrix),
    trainable=False,   # keep GloVe's vectors frozen instead of fine-tuning them
)

Learning embeddings inside a Keras model

You rarely train Word2Vec by hand anymore — most deep models simply start with a keras.layers.Embeddingkeras.layers.Embedding layer and let it learn word vectors as part of training:

import tensorflow as tf
 
vocab_size = 1000   # number of distinct words (+ OOV buckets)
embed_size = 16      # size of each word's dense vector
 
model = tf.keras.Sequential([
    # mask_zero=True tells every downstream layer to ignore padding tokens (ID 0)
    tf.keras.layers.Embedding(vocab_size, embed_size, mask_zero=True, input_shape=[None]),
    tf.keras.layers.GRU(32, return_sequences=True),
    tf.keras.layers.GRU(32),
    tf.keras.layers.Dense(1, activation="sigmoid"),
])
 
model.summary()

import tensorflow as tf
 
vocab_size = 1000   # number of distinct words (+ OOV buckets)
embed_size = 16      # size of each word's dense vector
 
model = tf.keras.Sequential([
    # mask_zero=True tells every downstream layer to ignore padding tokens (ID 0)
    tf.keras.layers.Embedding(vocab_size, embed_size, mask_zero=True, input_shape=[None]),
    tf.keras.layers.GRU(32, return_sequences=True),
    tf.keras.layers.GRU(32),
    tf.keras.layers.Dense(1, activation="sigmoid"),
])
 
model.summary()

The EmbeddingEmbedding layer’s output shape is [batch size, time steps, embed_size][batch size, time steps, embed_size] — it turns each 2D batch of word IDs into a 3D stack of word vectors, one per token.

Visualize it

Word embeddings live in a high-dimensional space, but if you squash them down to 2 dimensions (say, with PCA or t-SNE), similar words cluster together. Click to reshuffle and watch semantically related words stay near each other while unrelated clusters drift apart:

Geometric relationships, not just distances

Embeddings aren’t just “similar words are close together” — specific directions in the embedding space can carry meaning too. The classic example: the vector you’d add to go from “man” to “woman” is roughly the same vector that takes you from “king” to “queen” — a reusable “gender” direction. That means simple vector arithmetic can answer analogies:

import numpy as np
 
king  = np.array([0.90, 0.10, 0.40])
man   = np.array([0.85, 0.05, 0.35])
woman = np.array([0.75, 0.20, 0.45])
queen = np.array([0.85, 0.15, 0.42])   # the embedding the model actually learned
 
predicted_queen = king - man + woman
 
def cosine_sim(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
 
print("Predicted vector:", np.round(predicted_queen, 2))
print("Similarity to real 'queen' vector:", round(cosine_sim(predicted_queen, queen), 2))

import numpy as np
 
king  = np.array([0.90, 0.10, 0.40])
man   = np.array([0.85, 0.05, 0.35])
woman = np.array([0.75, 0.20, 0.45])
queen = np.array([0.85, 0.15, 0.42])   # the embedding the model actually learned
 
predicted_queen = king - man + woman
 
def cosine_sim(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
 
print("Predicted vector:", np.round(predicted_queen, 2))
print("Similarity to real 'queen' vector:", round(cosine_sim(predicted_queen, queen), 2))

Predicted vector: [0.8  0.25 0.5 ]
Similarity to real 'queen' vector: 0.99

Predicted vector: [0.8  0.25 0.5 ]
Similarity to real 'queen' vector: 0.99

Nobody tells the model that gender is a concept — a well-trained embedding space just ends up organized this way because “king” and “queen” (and “man” and “woman”) show up in structurally similar contexts.

Practical note

In modern NLP, embeddings are often learned as part of a deep model (Transformers). But Word2Vec/GloVe are great for understanding the concept.

Mini-checkpoint

What does it mean if two words have a high cosine similarity between embeddings?

• they appear in similar contexts and are semantically related.

🧪 Try It Yourself

Exercise 1 – Cosine Similarity Between Vectors

Exercise 2 – An Embedding Layer’s Output Shape

Exercise 3 – Analogies with Vector Arithmetic

Next

Continue to Sentiment Analysis Tutorial — put embeddings (or TF-IDF) to work on a real classification task.