What you’ll learn

• what NER is, and how it differs from a “one label per document” task like sentiment analysis
• the BIO tagging scheme used to label multi-word entities
• why NER is a sequence labeling problem — one prediction per token, not one per sentence
• the TimeDistributedTimeDistributed layer pattern for applying a classifier at every time step
• why padding tokens must be masked out of both training and evaluation

What NER does

Named Entity Recognition identifies and labels entities in text:

• PERSON — “Barack Obama”
• ORG — “OpenAI”
• GPE/LOC — “Paris”
• DATE — “January 31”

  flowchart LR
  subgraph Tokens["Input tokens"]
    direction LR
    W1["Barack"] --> W2["Obama"] --> W3["visited"] --> W4["Paris"]
  end
  Tokens --> M["Sequence model
(BiLSTM / Transformer)"]
  M --> Tags
  subgraph Tags["Predicted tags"]
    direction LR
    T1["B-PER"] --> T2["I-PER"] --> T3["O"] --> T4["B-LOC"]
  end

NER is sequence labeling, not classification

Sentiment analysis reads a whole review and outputs one label for the entire sequence — it’s a many-to-one problem. NER is different: it has to label every single token, so it’s a many-to-many problem. A four-word sentence needs four tags out.

That’s a meaningful architectural change. Instead of only keeping the RNN’s last output (as a sentiment model does), an NER model needs return_sequences=Truereturn_sequences=True on every recurrent layer, and it needs to run a classifier at every time step — not just the final one.

The BIO tagging scheme

Entities can span multiple tokens (“Barack Obama” is two words, one PERSON entity), so a plain per-token label isn’t quite enough — you also need to know where an entity starts versus continues. BIO tagging solves this with a simple prefix convention:

• B-<TYPE><TYPE> — the first (Beginning) token of an entity
• I-<TYPE><TYPE> — a token Inside an entity, continuing the previous one
• O — Outside any entity

Notice “Obama” is I-PERI-PER, not a fresh B-PERB-PER — the I-I- prefix is what tells you it’s a continuation of the same entity as the token before it, not a brand-new one.

Building the model: TimeDistributed for many-to-many output

The book’s approach to sequence-to-sequence prediction (originally shown for time series forecasting) applies directly here: keep return_sequences=Truereturn_sequences=True on every recurrent layer, then wrap the output classifier in a TimeDistributedTimeDistributed layer so it runs independently at every time step:

import tensorflow as tf
 
vocab_size = 5000
num_tags = 9   # e.g. O, B-PER, I-PER, B-ORG, I-ORG, B-LOC, I-LOC, B-DATE, I-DATE
 
model = tf.keras.Sequential([
    # mask_zero=True tells every downstream layer to ignore <pad> tokens (ID 0)
    tf.keras.layers.Embedding(vocab_size, 64, mask_zero=True, input_shape=[None]),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64, return_sequences=True)),
    tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(num_tags, activation="softmax")),
])
 
model.compile(loss="sparse_categorical_crossentropy", optimizer="adam", metrics=["accuracy"])
model.summary()

import tensorflow as tf
 
vocab_size = 5000
num_tags = 9   # e.g. O, B-PER, I-PER, B-ORG, I-ORG, B-LOC, I-LOC, B-DATE, I-DATE
 
model = tf.keras.Sequential([
    # mask_zero=True tells every downstream layer to ignore <pad> tokens (ID 0)
    tf.keras.layers.Embedding(vocab_size, 64, mask_zero=True, input_shape=[None]),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64, return_sequences=True)),
    tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(num_tags, activation="softmax")),
])
 
model.compile(loss="sparse_categorical_crossentropy", optimizer="adam", metrics=["accuracy"])
model.summary()

A few things worth noticing:

• BidirectionalBidirectional wraps the LSTM so it reads the sentence both forwards and backwards — knowing the word after “Paris” is just as useful for tagging it as knowing the word before.
• TimeDistributed(Dense(num_tags, "softmax"))TimeDistributed(Dense(num_tags, "softmax")) reshapes the sequence so the same DenseDense layer runs independently at every time step, turning each token’s hidden state into a probability distribution over tags.
• mask_zero=Truemask_zero=True on the EmbeddingEmbedding layer means padding tokens are automatically ignored by every downstream masking-aware layer — including Bidirectional(LSTM(...))Bidirectional(LSTM(...)).

Why masking matters here more than ever

Sentences in a batch rarely have the same number of tokens, so shorter ones get padded with a <pad><pad> token (ID 0) to match the longest one. Without masking, the model would happily learn to predict a tag for <pad><pad> — wasted capacity, and it can even drag down accuracy metrics, since a run of correctly predicted “O” tags for padding looks like free accuracy that has nothing to do with real language understanding. Masking makes sure padded positions contribute zero to both the loss and any evaluation metric.

Practical note

In Python, popular tools include spaCy (fast, production-ready, ships pretrained NER models out of the box) and Hugging Face Transformers (state-of-the-art accuracy via fine-tuned BERT-style models). You can learn the underlying concept — token classification with BIO tags — without installing either.

Mini-checkpoint

What’s the difference between NER and sentiment analysis, architecturally?

• Sentiment analysis is many-to-one: the model reads a whole sequence and outputs a single label. NER is many-to-many: it outputs one tag per input token, which is why it needs return_sequences=Truereturn_sequences=True and a TimeDistributedTimeDistributed classifier instead of a single final DenseDense layer.

🧪 Try It Yourself

Exercise 1 – BIO Tagging by Lookup

Exercise 2 – A Token Classification Model’s Output Shape

Exercise 3 – Masking Out Padding Tokens

Next

Continue to The Transformer Architecture — see how self-attention lets a model look directly at any token in the sequence instead of squeezing everything through a single recurrent hidden state, the idea behind BERT and GPT.