Evaluating Language Models (Perplexity and BLEU)

Every model in this phase has been scored with accuracy, which works because the answer was one of two labels. Generation has no such luxury: there are many acceptable outputs, most of them never written down, and the metric has to cope.

Two metrics dominate, and both are routinely misread. This page implements both, and the implementation is the point — perplexity gets compared across incomparable test sets, and BLEU gets quoted without the properties that make it fail.

What you’ll learn

Perplexity as exponentiated cross-entropy, and why uniform perplexity equals the vocabulary size.
The ladder on one shared test set: uniform 4,997 → unigram 408.05 → bigram 226.47 → neural 189.81.
Why perplexity numbers from two papers are usually not comparable.
Why BLEU gives a correct reordering and a correct paraphrase 0.0000.
The brevity penalty, alone, in a table.
What sampling temperature does to diversity and to likelihood — in opposite directions.

Perplexity

\text{PP} = \exp\!\left(-\frac{1}{N}\sum_{i=1}^{N} \log P(w_i \mid w_{<i})\right)

It is the exponential of the average negative log-likelihood, and it has a clean reading: the effective number of equally-likely choices the model faces at each step. A model with perplexity 200 is as uncertain as someone rolling a fair 200-sided die.

That reading gives a free sanity check. A model that has learned nothing assigns every word probability $1/V$ , so its perplexity is exactly $V$ — measured 4,997.00 on a 4,997-word vocabulary.

Model	Held-out perplexity	Tokens scored
uniform	4,997.00	30,000
unigram (add-0.1)	408.05	30,000
bigram (add-0.1)	226.47	30,000
neural (8-token context)	189.81	30,000

Two panels. Left: bars of held-out perplexity on a log scale — uniform 4,997.0, unigram 408.1, bigram 226.5 and neural 189.8. Right: the neural model's per-epoch validation perplexity, falling steeply to a minimum of 189.81 at epoch 5 and then rising to 256.49 by epoch 14, crossing back above the dashed bigram line. — The right panel is why the left panel says 189.81 rather than 256.49: this model's minimum is at epoch 5 of 14, and by the end it has overfitted past the bigram it beats. Reporting the final epoch instead of the best one — which an earlier version of this page did — turned a win into a loss of 4,062 perplexity.

The two ways this measurement goes wrong

Both happened here before the numbers above were trustworthy.

Scoring different test sets. The first run scored the n-gram models on all 231,210 held-out tokens and the neural model on 30,000 windows, then printed them in one table. Perplexity is an average over whatever you scored, so that comparison was meaningless. Every model above sees the identical windows.
Reporting the final epoch. With 1,030,085 parameters — most of them in the output layer — the model overfits hard. Final-epoch perplexity was 4,062.44 in one configuration; the minimum was far lower.

And a third that cannot be fixed by being careful, only by disclosure: perplexity depends on the tokenizer. A model over 2,070 BPE pieces and a model over 39,563 words are averaging over different units, so their perplexities are not on the same scale. Comparing them is like comparing prices without saying which currency.

BLEU

BLEU compares an output against one or more references by counting matching n-grams:

\text{BLEU} = \underbrace{\text{BP}}_{\text{brevity penalty}} \cdot \exp\!\left(\frac{1}{N}\sum_{n=1}^{N} \log p_n\right), \qquad \text{BP} = \min\!\left(1,\ e^{1 - r/c}\right)

where $p_n$ is the clipped precision of $n$ -grams and $r/c$ is the reference length over the candidate length. Measured against the reference 'the cat sat on the mat':

Candidate	BLEU-4	BP	$p_1$	$p_2$	$p_3$	$p_4$
exact match	1.0000	1.000	1.000	1.000	1.000	1.000
one word changed	0.5373	1.000	0.833	0.600	0.500	0.333
reordered	0.0000	1.000	1.000	0.800	0.500	0.000
shorter but correct	0.0000	0.368	1.000	1.000	1.000	0.000
synonym	0.5373	1.000	0.833	0.600	0.500	0.333
padded with repeats	0.6804	1.000	0.750	0.714	0.667	0.600

Two panels. Left: horizontal bars of BLEU-4 for six candidate sentences against one reference, with exact match at 1.0, one-word-changed and synonym at 0.5373, padded-with-repeats at 0.6804, and both reordered and shorter-but-correct at zero. Right: BLEU against maximum n-gram order for three candidates, where the reordered sentence scores well at order 1 and collapses to zero at order 4. — Two zeros to explain. 'On the mat the cat sat' contains every reference unigram — p1 is a perfect 1.000 — and still scores 0.0000, because the geometric mean over precisions is zero as soon as any one of them is, and no 4-gram survives the reordering. 'The cat sat' has perfect precision at every order and is destroyed by a brevity penalty of 0.368. Meanwhile 'the cat sat on the mat mat mat' — padded with nonsense — scores 0.6804, higher than either.

Read that last row again: a sentence padded with repeated words outscored a correct shorter one by 0.68. BLEU is a precision-based metric with a length hack bolted on, and its failures are structural rather than incidental:

It compares surface strings. feline for cat scores exactly as badly as a wrong word.
A single zero precision zeroes everything, because the combination is a geometric mean. Real implementations apply smoothing for this reason — usually add-1 on the counts.
The brevity penalty is the only thing preventing degenerate short outputs, and it is asymmetric: too long is free, too short is punished exponentially.

Candidate length	Reference length	BP
3	6	0.3679
4	6	0.6065
5	6	0.8187
6	6	1.0000
7	6	1.0000
12	6	1.0000

BLEU is still used because on corpus-level translation with multiple references it correlates reasonably with human judgement. On single sentences with one reference — the way it is quoted in most blog posts — it is close to noise.

Temperature: diversity against likelihood

Sampling divides the logits by a temperature before the softmax. Measured on 2,000 predictions from the trained model:

Temperature	Distinct tokens	Mean top probability	Entropy	Perplexity
0.2	69	0.8420	0.4265	98,041.8
0.5	174	0.5845	1.5489	1,162.6
0.8	493	0.3016	3.5904	283.5
1.0	714	0.1788	4.7475	225.5
1.5	1,044	0.0580	6.3151	243.6

Two panels. Left: distinct tokens produced against sampling temperature, rising from 69 at 0.2 to 1,044 at 1.5. Right: perplexity of the reweighted distribution on a log scale, falling from 98,041.8 at temperature 0.2 to a minimum of 225.5 at 1.0 and rising slightly to 243.6 at 1.5. — The perplexity minimum sits at temperature 1.0, which is not a coincidence: that is the distribution the model was trained to produce, and any reweighting makes it a worse predictor of the actual next token. Sharpening to 0.2 raises perplexity by a factor of 435 while cutting the vocabulary in use from 714 tokens to 69 — the numerical signature of repetitive output.

Which is the honest limitation of perplexity for generation: it is minimised at temperature 1.0, but almost nobody generates at temperature 1.0, because samples from the raw distribution wander. The setting people actually ship — 0.7–0.9, or top-p sampling — is a setting perplexity says is worse.

diagram Diagram mermaid

  flowchart TB
  A["what are you evaluating?"] --> B{"is there one correct output?"}
  B -->|"yes — classification"| C["accuracy, F1"]
  B -->|"no — generation"| D{"do you have references?"}
  D -->|"no"| E["perplexity
same tokenizer, same test set"]
  D -->|"yes, several"| F["BLEU / chrF / METEOR
corpus level, with smoothing"]
  D -->|"yes, one"| G["BLEU is close to noise here
— use human judgement or a learned metric"]
  E --> H["report the best epoch and
the tokenizer"]
  F --> H

sketch Perplexity as effective choices p5.js

Move the probability the model assigns to the true next word and watch perplexity. Perplexity is the size of the fair die that would be equally uncertain.

sketch The measured table, ranked p5.js

Click a column to rank every row by it. The bars are that column's values and the highest and lowest are computed from the numbers, not written in.

Pitfalls

Comparing perplexities across tokenizers. 2,070 BPE pieces and 39,563 words are different units; the numbers are not on one scale.
Comparing perplexities across test sets. Scoring 231,210 tokens against 30,000 windows produced a table that meant nothing.
Reporting the final epoch. The neural model’s best was 189.81 at epoch 5 and 256.49 at epoch 14 — and 4,062.44 in an unregularised configuration.
Quoting BLEU on single sentences with one reference. A correct reordering scores 0.0000 and a padded sentence scores 0.6804.
Using BLEU-4 without smoothing. One zero precision zeroes the whole score.
Forgetting the brevity penalty is asymmetric. Twelve tokens against six is free; three against six costs a factor of 0.368.
Treating a perplexity win as a generation win. Perplexity is minimised at temperature 1.0, which is not the temperature anyone generates at.

Recap

Perplexity is exponentiated cross-entropy — the effective number of equally-likely choices per step — and a model that learned nothing scores exactly the vocabulary size (4,997.00).
On one shared 30,000-window test set: unigram 408.05, bigram 226.47, neural 189.81.
The neural model’s minimum was at epoch 5 of 14 (256.49 at the end): report the best epoch.
BLEU gives a correct reordering 0.0000 and a correct paraphrase 0.5373, while a repeat-padded sentence scores 0.6804.
The brevity penalty runs 0.3679 at half length and 1.0000 at any length above the reference.
Temperature 0.2 cut distinct tokens from 714 to 69 and raised perplexity 435×; perplexity is minimised at exactly 1.0.

That closes the NLP phase. Generation moves from measuring text to producing images and sound: Phase 6 — Generative Deep Learning.

pch.quizTag pch.quizDefaultTitle

A model with a 4,997-word vocabulary that has learned nothing scores perplexity 4,997.00. Why exactly the vocabulary size?

pch.quizShowAnswer

B — It assigns every word probability 1/V, so exponentiated cross-entropy is exactly V — perplexity is the effective number of equally-likely choices per step — That gives a free sanity check: any perplexity at or above the vocabulary size means the model has learned nothing usable.
The neural model scored perplexity 189.81 at epoch 5 and 256.49 at epoch 14. Which do you report, and why does it matter?

pch.quizShowAnswer

B — The best epoch — with 1,030,085 parameters the model overfits, and an earlier unregularised run reported 4,062.44 at the final epoch while its minimum was far lower — Reporting the final epoch turned a win over the bigram into an apparent loss of thousands of perplexity points.
'On the mat the cat sat' contains every unigram of 'the cat sat on the mat' and scores BLEU-4 = 0.0000. Why?

pch.quizShowAnswer

B — BLEU combines n-gram precisions with a geometric mean, so a single zero — here p4, since no 4-gram survives the reordering — zeroes the whole score — This is why real implementations smooth the counts. The same table shows a repeat-padded sentence scoring 0.6804 — higher than a correct shorter one.
Why are two published perplexity numbers usually not comparable?

pch.quizShowAnswer

B — Because perplexity is an average per token, so it depends on the tokenizer and on the test set — a model over BPE pieces and one over whole words are averaging different units — The subword page measured 2,070 pieces against 39,563 words on the same corpus — the same text, very different token counts.
Perplexity of the reweighted distribution was minimised at temperature 1.0 (225.5) and rose to 98,041.8 at 0.2. What does that tell you about using perplexity to tune generation?

pch.quizShowAnswer

B — Perplexity is minimised at exactly the distribution the model was trained on, so it cannot recommend the sharper settings people actually ship — it measures prediction, not sample quality — Sharpening to 0.2 cut the vocabulary in use from 714 tokens to 69 — the numerical signature of repetition, which perplexity registers only as 'worse'.

🧪 Try It Yourself

pch.coffeeTagline

pch.coffeeCta

pch.feedbackHeading

pch.feedbackSubheading