Named Entity Recognition (NER)

Named entity recognition tags each token with the span it belongs to: mary/B-PER okafor/I-PER joined/O acme/B-ORG industries/I-ORG. It looks like classification with more labels, and that resemblance is exactly what makes it easy to measure wrongly.

The best model on this page scores 0.8011 token accuracy and 0.4389 entity F1 on the same predictions. Both numbers are correct. Only one of them is the thing you care about.

What you’ll learn

BIO tagging, and how a tag sequence becomes a set of spans.
Why token accuracy flatters: predicting O everywhere already scores 0.5653.
Why an entity counts only if its start, end and type are all right.
The measurement that separates a real tagger from a word list: 38.00% of test tokens are words the model has never seen.
What the errors actually are — 48.09% missed, 17.04% boundary-off — and what that implies about the fix.

The corpus, and why it is generated

CoNLL-2003 is not redistributable, so this page generates its data: templated sentences with PERSON, LOCATION and ORGANISATION spans and exact BIO tags. Generated data gets the labels right by construction, and it lets the difficulty be set deliberately rather than inherited.

The first version of this corpus was useless. Every model scored a perfect 1.0000 token accuracy and 1.0000 entity F1, because the entity pools were small closed lists — the models memorised which strings are names and never had to look at context. That measures a lookup table.

Two changes fix it, and they are the two things that make real NER hard:

The test entity pools are disjoint from the training ones. Every test name, city and organisation is a string the model has never seen — 38.00% of test tokens are unknown words.
Some words are ambiguous. may, hope, mark, summit, delta and orion appear both as entity words and as ordinary filler. No lookup table resolves them; only the surrounding context can.

Token accuracy is the wrong metric

Tag	Tokens	Share
O	11,469	0.5653
B-PER	1,500	0.0739
I-PER	1,500	0.0739
B-LOC	1,500	0.0739
I-LOC	977	0.0482
B-ORG	1,500	0.0739
I-ORG	1,842	0.0908

A model that predicts O for every token scores 0.5653 token accuracy and finds zero entities. On real newswire the O share is closer to 0.83, so the free score is higher still.

figure 1,500 held-out sentences matplotlib

The right panel is the page in one picture. Every model's token accuracy is comfortably above the 0.5653 floor and looks respectable; the entity F1 next to it tells a completely different story, and the ordering of the two metrics is not even the same. The per-token dense model beats the do-nothing baseline by 0.0097 on tokens while finding almost nothing: F1 0.0317.

Spans, not tokens

An entity is correct only if its start, end and type all match:

BIO tags to spans

def spans(tags):
    out, start, kind = set(), None, None
    for position, tag in enumerate(tags):
        if tag.startswith("B-"):
            if start is not None:
                out.add((start, position, kind))
            start, kind = position, tag[2:]
        elif tag.startswith("I-") and start is not None and tag[2:] == kind:
            continue                      # the span continues
        else:
            if start is not None:
                out.add((start, position, kind))
            start, kind = None, None
    if start is not None:
        out.add((start, len(tags), kind))
    return out

Getting acme/B-ORG industries/O right on one token out of two is not half an entity — it is a missed entity and a spurious one-token entity. That asymmetry is why entity F1 sits so far below token accuracy.

What the three models did

Model	Parameters	Token accuracy	Precision	Recall	Entity F1
all `O`	0	0.5653	0.0000	0.0000	0.0000
per-token dense	5,959	0.5750	0.2594	0.0169	0.0317
1D convolution	10,055	0.8011	0.5921	0.3487	0.4389
bidirectional LSTM	20,487	0.7845	0.2152	0.2309	0.2228

Grouped bars showing token accuracy, entity precision, entity recall and entity F1 for four models. The all-O baseline has token accuracy 0.57 and zeros elsewhere; per-token dense has high token accuracy but near-zero recall; the 1D convolution leads on every entity metric; the bidirectional LSTM sits between them. — The per-token dense model is the control that makes the point: with no access to neighbouring words it cannot tell an unseen name from an unseen filler word, so it scores 0.2594 precision at 0.0169 recall — it tags almost nothing, and token accuracy still reads 0.5750. Both convolution and recurrence can see context; the convolution's fixed 3-token window happened to suit this template-generated data better than the LSTM's unbounded one at this budget.

Two honest observations:

The per-token dense model fails by construction. It sees one embedding at a time. Since 38% of test entity words are unknown tokens, it has nothing to go on — and its 0.5750 token accuracy still looks like a working model.
The bidirectional LSTM lost to a 1D convolution (0.2228 against 0.4389). The entities here sit inside rigid templates where the informative context is one or two words away, which is precisely a convolution’s window. This is a property of the generated corpus, not a general result — on real newswire, where entity context is longer-range and messier, recurrent and transformer taggers win comfortably.

Where the errors are

A bar chart of entity-level outcomes for the 1D convolution: 1,569 exact at 34.87%, 0 wrong type, 767 boundary-off at 17.04%, 2,164 missed at 48.09%, and 0 spurious. — Nearly half the entities are missed outright — unsurprising when the words are unseen. The 17.04% boundary-off cases are the interesting ones: the model found the entity and got its type right but its start or end wrong, usually by dropping the first token of a two-word name. Those are entirely invisible to token accuracy, which happily credits the token it did get right.

Outcome	Count	Share
exact	1,569	34.87%
wrong type	0	0.00%
boundary off	767	17.04%
missed	2,164	48.09%
spurious	0	0.00%

A worked example from the test set:

text

truth:     coastal/B-ORG power/I-ORG   rashid/B-PER moreau/I-PER   oslo/B-LOC
predicted:               power/I-ORG                moreau/I-PER   oslo/B-LOC

Two spans lost their first token. Under strict BIO decoding an I- tag with no B- before it opens nothing at all, so those two entities do not become shortened entities — they disappear. Token accuracy still counts five of seven tokens right (0.7143); the entity score for that sentence is one span out of three. The model has learned “the second word of a name is part of a name” without learning where names begin, and only the span-level metric can see it.

That failure mode is exactly what a CRF layer on top of the tagger is for: it scores whole tag sequences rather than independent tokens, so I-PER following O — which is not a legal BIO sequence at all — can be ruled out structurally instead of hoped for.

diagram Diagram mermaid

  flowchart LR
  A["tokens"] --> B["embedding"]
  B --> C{"context?"}
  C -->|"none — per-token dense"| D["F1 0.0317
cannot see neighbours"]
  C -->|"3-token window — Conv1D"| E["F1 0.4389"]
  C -->|"whole sentence — BiLSTM"| F["F1 0.2228"]
  E --> G["per-token softmax"]
  F --> G
  G --> H["illegal sequences possible
e.g. O then I-PER"]
  H --> I["a CRF scores whole sequences
and rules them out"]

sketch Tags to spans, and what counts as correct p5.js

Click a token to cycle its predicted tag. The span-level score updates — and only exact matches count.

sketch The measured table, ranked p5.js

Click a column to rank every row by it. The bars are that column's values and the highest and lowest are computed from the numbers, not written in.

Pitfalls

Reporting token accuracy. All-O scores 0.5653 here and 0.83+ on real newswire.
Building a corpus where entity strings repeat between train and test. Every model scored 1.0000 until the pools were made disjoint — the benchmark was a word list.
Ignoring boundary errors. 17.04% of entities were found with the right type and the wrong span; token accuracy credits them.
Assuming a bigger model wins. The bidirectional LSTM has twice the parameters of the convolution and scored half the F1 on this data.
Using a per-token softmax and expecting legal sequences. O followed by I-PER is not valid BIO; only a CRF or constrained decoding rules it out structurally.
Micro-averaging over a corpus with one dominant type. Report per-type F1 as well, or a large class hides the others.
Generalising from generated data. These templates favour a short context window; real newswire does not.

Recap

BIO tags become spans; an entity is correct only if start, end and type all match.
Predicting O everywhere scored 0.5653 token accuracy and 0.0000 entity F1.
Best model: 1D convolution at 0.8011 token accuracy but 0.4389 entity F1.
A per-token dense model — no context — reached 0.5750 token accuracy and 0.0317 F1.
38.00% of test tokens were unseen words, which is what forces the model to use context instead of memorising strings.
Errors: 48.09% missed, 17.04% boundary-off, and the worked example lost the first token of every span.

Tagging reads a sequence and labels it in place. Generating a different sequence needs a decoder, and that is the next architecture: The Transformer Architecture.

pch.quizTag pch.quizDefaultTitle

A tagger scores 0.8011 token accuracy and 0.4389 entity F1 on the same predictions. Which should you report, and why?

pch.quizShowAnswer

B — Entity F1 — an entity is only useful if its start, end and type are all correct, and predicting O everywhere already scores 0.5653 token accuracy while finding nothing — On real newswire the O share is around 0.83, so the free token-accuracy score is higher still.
The first version of this corpus gave every model 1.0000 token accuracy and 1.0000 entity F1. What was wrong?

pch.quizShowAnswer

B — The entity pools were small closed lists shared between train and test, so the models memorised which strings are names and never used context — the benchmark measured a lookup table — Making the test pools disjoint pushed 38% of test tokens to unknown words, and the scores immediately became informative.
A per-token dense model scored 0.5750 token accuracy but only 0.0317 entity F1. Why is it so bad at entities?

pch.quizShowAnswer

B — It sees one token embedding at a time with no neighbours, and 38% of test entity words are unknown — so it has no information to distinguish an unseen name from unseen filler — It is the control that proves context is doing the work in the other two models — and its token accuracy still sits above the all-O baseline.
17.04% of entities were 'boundary off' — right type, wrong span. Why does token accuracy hide these?

pch.quizShowAnswer

B — Because it credits every token the model got right, so a two-word name tagged on only its second token still contributes a correct token while the entity is entirely lost — The worked example lost the first token of all three spans: 0.625 token accuracy, one correct span out of three.
What problem does a CRF layer on top of a tagger solve?

pch.quizShowAnswer

B — A per-token softmax can emit illegal sequences such as O followed by I-PER; a CRF scores whole tag sequences, so invalid transitions are ruled out structurally — That directly attacks the boundary errors, which were 17.04% of all entities here.

🧪 Try It Yourself

pch.coffeeTagline

pch.coffeeCta

pch.feedbackHeading

pch.feedbackSubheading