Phase 5 - NLP & Transformers

Text is not numbers, and every step from one to the other throws something away. This phase measures each step: which words to keep, how to split them, what “similar” means, and what attention buys over recurrence.

Two results frame everything else. A linear model over word counts beat every neural model here — 0.8632 against a transformer’s 0.8462, in 8.4 seconds against 460. And on a task that needs two composed operations, one attention block scored 0.3120 while two scored 0.9990. Neither is a slogan; both are measurements, and they point in opposite directions about when architecture matters.

The pages

#	Page	The measured headline
441	Text Preprocessing	lowercasing this corpus changes 0 words — it was pre-processed; br is the 7th most frequent token
441.5	Subword Tokenization	BPE at 2,070 pieces reaches OOV 0.0000 against 0.0293 for 20,000 words
442	Bag of Words & TF-IDF	sublinear TF-IDF 0.8748 in 0.4 s; TF-IDF at 2,000 features beat counts at 30,000
443	Word Embeddings	sentiment-trained vectors collapse to cosine 1.000; skip-gram finds movie/film at 0.746
444	Sentiment Analysis	five families: TF-IDF 0.8632 wins, transformer 0.8462 at 55× the cost
445	Named Entity Recognition	token accuracy 0.8011, entity F1 0.4389 — and all-O scores 0.5653
445.5	Attention from Scratch	one block 0.3120, two blocks 0.9990; unscaled attention peaks at 0.9355
446	The Transformer Architecture	every ablation helps at 1 block; removing layer norm costs 0.0612 at 4
447	Sequence-to-Sequence	attention took exact match 0.6250 → 0.9770 for 768 parameters
447.5	Evaluating Language Models	neural 189.81 vs bigram 226.47; BLEU gives a correct reordering 0.0000

diagram Diagram mermaid

  flowchart TD
  A["441 tokens
Zipf, Heaps, OOV"] --> B["441.5 subwords
BPE, zero OOV"]
  B --> C["442 counts
BoW, TF-IDF"]
  C --> D["443 embeddings
dense, learned"]
  D --> E["444 the bake-off
five families"]
  D --> F["445 tagging
spans, not tokens"]
  E --> G["445.5 attention
Q, K, V"]
  G --> H["446 the block
norms, residuals"]
  H --> I["447 encoder-decoder
teacher forcing"]
  I --> J["447.5 evaluation
perplexity, BLEU"]

figure IMDB, 8,000 reviews, 6 epochs, matched width matplotlib

The transformer's entire learning happens in epoch 1 — 0.8462, which is also its best score across all six epochs. Attention over 200 positions in parallel means every token pair is one gradient step apart, so there is no long credit-assignment chain to unroll. That is the real advantage on display here; it is not accuracy, since the bag of words matched it at 0.8465 in a tenth of the time.

figure 200 random query-key draws, 12 tokens matplotlib

Without the scale, attention at d_k = 256 puts 0.9355 of its weight on a single token before any training — the softmax has saturated on noise, and its gradient with it. The scaled version holds near 0.28 at every dimension, which is the point: the operation should behave the same whether the head is 4-dimensional or 256-dimensional. The entropy panel says the same thing in nats, against a uniform maximum of 2.4849.

figure Every claim this phase set out to test matplotlib

Collected from the runs behind each page's own figures rather than measured afresh, so every bar is traceable to the page it names. Bar length is the log of the effect size, because the effects span from 0.0014 to 5,376 — the number that matters is printed on each bar. Across all nine phases, 34 of 54 claims were contradicted outright, 11 held at a cost that was worth stating, and 9 held as advertised.

Four results that contradict the usual story

1. The simplest model won. TF-IDF with logistic regression scored 0.8632 against an average embedding’s 0.8575, a convolution’s 0.8495, a transformer’s 0.8462 and an LSTM’s 0.8415 — with 9,768 parameters and 8.4 seconds. Sentiment is close to a keyword task, and 8,000 reviews is not much data.
2. A transformer block’s components do not help a shallow transformer. At one block, removing the feed-forward network, the residuals or the layer norms all raised accuracy. Layer normalisation only earns its 128 parameters at depth, where removing it costs 0.0612.
3. Sentiment-trained embeddings collapse. terrible, poor, waste and bad end up at cosine 1.000 — parallel vectors — because a one-dimensional loss needs exactly one direction. The same corpus trained on co-occurrence gives movie/film at 0.746.
4. Attention’s win is convergence, not accuracy — until composition is needed. The transformer matched a bag of words (−0.0002) but reached its best score in epoch 1. On the induction task, where the model must match a cue and shift one position, one block cannot do it at all.

Four corpora that were too easy, and what fixing them showed

Three modules in this phase produced perfect or meaningless scores on the first attempt. All three are now rebuilt, and the fixes are the most transferable content here.

Experiment	First attempt	The flaw	After fixing
NER	every model 1.0000	entity strings shared between train and test — the benchmark measured a word list	test pools disjoint, 38% unknown tokens → best F1 0.4389
seq2seq	everything 1.0000	a bare date is trivially learnable	date buried in filler → attention worth +0.3520 exact match
skip-gram	accuracy 0.4585 (below chance)	positives built first, so validation_split took only negatives	shuffled → 0.6470, and neighbours became semantic
perplexity	n-grams and neural in one table	scored on different test sets (231,210 tokens vs 30,000 windows)	same windows → neural 189.81 beats bigram 226.47

A fifth was a metric that flattered rather than a corpus that was easy: reporting the final epoch’s perplexity instead of the best gave 4,062.44 where the minimum was 189.81.

sketch How often the standard story survived p5.js Pause Restart

Step through the phases. Each bar splits the claims that phase tested into contradicted, held at a price, and held as advertised - the totals are summed live.

The habit this phase teaches

Every page compares against something that costs nothing:

Task	The do-nothing baseline	What it exposed
vocabulary choice	coverage at each cap (0.7624 at 1,000 words)	23.8% of the text discarded silently
classification	TF-IDF + logistic (0.8632)	four neural families that never beat it
tagging	predict O everywhere (0.5653)	a model at 0.5750 token accuracy finding almost nothing
language modelling	uniform = vocabulary size (4,997.00)	whether a perplexity is meaningful at all
generation	greedy exact match, not per-character	0.9217 character accuracy = 0.6250 exact

sketch Accuracy against what it cost p5.js Pause Restart

Five model families on the same 8,000 IMDB reviews. Drag to weight training time; the ranking re-sorts and the winner changes.

What carries into the rest of the module

Idea from this phase	Where it comes back
attention over a set of tokens	diffusion U-Nets, every modern generative model
pre-norm blocks with residuals	Phase 6’s architectures, Phase 8’s scaling
teacher forcing and exposure bias	autoregressive image and audio generation
perplexity and its tokenizer dependence	Phase 6’s text generation page
“measure against the do-nothing baseline”	every remaining phase

From the book

Chollet introduces text vectorisation by noting that deep learning for text is “pattern recognition applied to words, sentences and paragraphs” — not understanding. The bake-off on page 444 is that sentence as a measurement: on a task decidable from word presence, a model that cannot represent word order at all scored highest.

From the book

The attention paper’s title claims attention is all you need. Page 446 takes the claim apart component by component and finds that at one block it is more than you need — and that the pieces which look redundant there (layer normalisation especially) are exactly the pieces that stop a four-block stack from degrading. The architecture is a depth-enabling device, and its parts only make sense at depth.

pch.quizTag pch.quizDefaultTitle

1. TF-IDF with logistic regression scored 0.8632 in 8.4 seconds against a transformer's 0.8462 in 460. What is the defensible conclusion?

   A Transformers do not work for sentiment B On 8,000 reviews of a near-keyword task, the extra machinery has nothing to learn that counts do not already capture - which is a claim about this budget and this task, not about transformers C The transformer was implemented incorrectly D Accuracy is the wrong metric
   pch.quizShowAnswer

   B — On 8,000 reviews of a near-keyword task, the extra machinery has nothing to learn that counts do not already capture - which is a claim about this budget and this task, not about transformers

2. One attention block scored 0.3120 on the induction task while two scored 0.9990. Why can one block not do it?

   A One block has too few parameters B The task needs two composed operations - match a cue, then shift one position - and a single attention layer can only perform one round of look-up before the output is read C One block trains too slowly D The softmax saturates
   pch.quizShowAnswer

   B — The task needs two composed operations - match a cue, then shift one position - and a single attention layer can only perform one round of look-up before the output is read

3. Embeddings trained on the sentiment objective put `terrible`, `poor`, `waste` and `bad` at cosine 1.000 - exactly parallel. What caused that?

   A A bug in the cosine calculation B A one-dimensional loss needs exactly one direction, so every word useful for the task collapses onto that axis and all other structure is discarded C Too few training epochs D The vocabulary was too small
   pch.quizShowAnswer

   B — A one-dimensional loss needs exactly one direction, so every word useful for the task collapses onto that axis and all other structure is discarded

4. NER token accuracy was 0.8011 while entity F1 was 0.4389, and labelling everything `O` already scores 0.5653 on tokens. What is the lesson?

   A The tagger is broken B Token accuracy on a task dominated by one class is close to a do-nothing baseline; the unit you evaluate has to be the unit you care about, which for NER is the whole span C F1 is too harsh D The dataset is too small
   pch.quizShowAnswer

   B — Token accuracy on a task dominated by one class is close to a do-nothing baseline; the unit you evaluate has to be the unit you care about, which for NER is the whole span

5. At one transformer block, removing the feed-forward network, the residuals or the layer norms all RAISED accuracy; at four blocks, removing layer norm cost 0.0612. What does that pattern say?

   A The components are useless B Those components solve problems that only appear with depth, so a shallow ablation measures their cost without ever exposing their benefit C The four-block model was overfitting D Layer norm should always be removed
   pch.quizShowAnswer

   B — Those components solve problems that only appear with depth, so a shallow ablation measures their cost without ever exposing their benefit