Capstone 2 - A Text Classifier End to End

Capstone 1 built a ladder that climbed. This one does not, and the point of the project is to notice that before spending three days on the top rung.

IMDB sentiment, 6,000 training reviews and 3,000 held out, every model given 6 epochs:

Model	Parameters	Seconds	Accuracy	Gain
Always predict one class	0	0	0.5220	—
Bag of words + linear	10,001	3	0.8507	+0.3287
Bag of words + MLP	160,033	4	0.8567	+0.0060
Embedding + pooling	321,089	6	0.8587	+0.0020
LSTM	329,409	53	0.8353	−0.0233
Bidirectional LSTM	338,753	113	0.7887	−0.0467
Self-attention	329,505	165	0.7957	+0.0070

A linear model over a multi-hot vector — no word order, no sequence information at all — reached 0.8507 in three seconds. Every recurrent and attention-based model, at 18× to 55× the training time, did worse.

What you’ll learn

Why the sequence models lost here, and what would have to change for them to win.
How to test the specific claim recurrence makes, rather than the general one.
Why the best single model is not the best available accuracy on this data.
What to report when your ladder stops climbing.

The ladder that stops

Left: horizontal bars of accuracy for seven models. The constant baseline is at 0.5220, bag of words at 0.8507, embedding-plus-pooling highest at 0.8587, then LSTM at 0.8353, bidirectional LSTM at 0.7887 and self-attention at 0.7957. Right: accuracy against training time on a log axis, showing the cheapest models at the top left and the most expensive at the bottom right. — The right panel is the summary: accuracy falls as training time rises. The bag-of-words model sits at the top left — 3 seconds, 0.8507 — and self-attention at the bottom right at 165 seconds and 0.7957. Nothing after rung three earns its cost, and two rungs actively lose ground.

The honest reading is not “recurrence does not work for sentiment”. It is more specific and more useful:

At 6,000 reviews and 6 epochs, a bag of words is the right model. Sentiment classification is substantially a matter of which words appear — terrible, superb, waste — and that is exactly what a multi-hot vector encodes. Word order carries additional signal, but extracting it requires learning an embedding and a sequence function from the same 6,000 examples, and there is not enough data to do both.

Three things would change the answer, and none of them are architectural:

More data. The published results that make LSTMs look good on IMDB use 25,000 training reviews, roughly 4× what is used here.
More epochs. The recurrent models are still improving at 6 epochs while the bag of words has converged; they are undertrained, not incapable.
Pretrained embeddings. The sequence models are learning word meanings from scratch, which is the expensive part and the part self-supervised pretraining exists to remove.

Stating those explicitly is part of the deliverable. A result of “the LSTM lost” without them is a claim about LSTMs; with them it is a claim about this budget.

Test the specific claim

Recurrence claims to use word order, which should matter most where a bag of words has the most to lose: long reviews, where the same words could support either sentiment depending on arrangement. So split the test set by true review length:

Grouped bars of accuracy across four review-length buckets for four models. In every bucket the bag-of-words and pooling models lead, and the gap widens on the longest reviews, where bag of words scores 0.8347 against the LSTM's 0.7813 and self-attention's 0.7387. — This is the panel that settles it. On reviews of 400 or more tokens — where word order should be doing the most work — the bag of words scores 0.8347 and the LSTM 0.7813. The sequence models lose by more on long reviews than on short ones, which is the opposite of the prediction. Note also that 43.3% of reviews exceed the 200-token window, so all of these models see a truncated version of the longest ones.

Model	0–100	100–200	200–400	400+
Bag of words + linear	0.8618	0.8550	0.8457	0.8347
Embedding + pooling	0.8672	0.8628	0.8563	0.8400
LSTM	0.8428	0.8515	0.8292	0.7813
Self-attention	0.8157	0.8067	0.7939	0.7387

Every model degrades on long reviews, which is expected — 43.3% of reviews are truncated at the 200-token window, so the longest ones are being judged on their first half. But the sequence models degrade fastest, and self-attention worst of all at 0.7387.

A recurrent model has to carry information across more timesteps on a long review, and with 6,000 examples it has not learned to do that reliably. The bag of words has no such problem: its representation of a 400-token review is exactly as easy to classify as its representation of a 50-token one.

diagram Diagram mermaid

  flowchart TD
  R["6,000 IMDB reviews"] --> B["bag of words
0.8507 in 3s"]
  R --> S["sequence models
0.7887 to 0.8353 in 53-165s"]
  B --> Q{"why did the simple
model win?"}
  S --> Q
  Q --> D["not enough data to learn
embeddings AND a sequence function"]
  Q --> T["43.3% of reviews truncated
at the 200-token window"]
  Q --> E["6 epochs: the recurrent models
are still improving"]

The best model is not the best accuracy

The two families disagree on 8.2% of reviews, and they are not the same 8.2%:

Left: bars showing 2,441 reviews both models got right, 111 only the simple model, 135 only the pooling model, and 313 both wrong. Right: three bars comparing the simple model at 0.8507, the pooling model at 0.8587 and an oracle that picks the better model per review at 0.8957. — 313 reviews defeat both models and 246 are solved by exactly one of them. An oracle that chose the right model for each review would score 0.8957 — 0.0370 above the better single model. That gap is not achievable directly, but it is the measurement that justifies an ensemble, and it is invisible if you only compare the two accuracy numbers.

Outcome	Reviews	Share
Both right	2,441	81.4%
Only bag of words	111	3.7%
Only embedding + pooling	135	4.5%
Both wrong	313	10.4%
Oracle (pick the better one)	—	0.8957

The 10.4% both models fail is the genuinely hard remainder — sarcasm, mixed reviews, reviews about a film’s subject rather than its quality. No amount of architecture search moves those; better data or a fundamentally stronger model would.

The 8.2% one model gets and the other misses is the opportunity. Averaging the two models’ probabilities is a two-line change and captures part of that 0.0370.

What to hand in

The full ladder including the rungs that lost. “We tried an LSTM and it scored 0.8353” is a result; omitting it is not.
The budget beside every number. 6,000 reviews, 6 epochs, 200-token window, 43.3% truncated. Every conclusion here is conditional on those.
One test of the specific claim, not just overall accuracy. The length breakdown is what turns “the LSTM lost” into “the LSTM lost by more where it should have won”.
The disagreement analysis, with the oracle number.
What you would change, with reasons: more data, more epochs, pretrained embeddings — in that order, because that is the order of expected effect.

sketch Accuracy against what it cost p5.js

Drag the slider to set how much you value training time. The ranking re-sorts, and the winner changes.

sketch The measured table, ranked p5.js

Click a column to rank every row by it. The bars are that column's values and the highest and lowest are computed from the numbers, not written in.

Pitfalls

Assuming the fancier model wins. Six seconds of bag of words beat 165 seconds of self-attention by 0.0550.
Reporting only the winning rung. The negative gains are the finding.
Concluding “LSTMs are bad at sentiment”. They are bad at sentiment with 6,000 examples and 6 epochs, which is a different claim.
Ignoring truncation. 43.3% of reviews exceed the window, so every model reads only part of the longest ones.
Comparing accuracy without comparing cost. 0.0080 of accuracy for 55× the training time.
Stopping at the accuracy numbers. The oracle at 0.8957 is 0.0370 above the best single model and only visible from the disagreement table.
Using mask_zero=False with padded sequences. The model then spends capacity modelling padding, which is pure loss on a dataset where 43.3% of rows are padded or truncated.

Recap

The ladder stopped at rung three: bag of words 0.8507, pooling 0.8587, then three sequence models at 0.7887–0.8353.
A linear model with no word-order information at all reached 0.8507 in 3 seconds.
The sequence models lost by more on long reviews (0.7387 for attention at 400+ tokens), the opposite of what recurrence predicts.
43.3% of reviews are truncated at 200 tokens, which limits every model on this page.
The two families disagree on 8.2% of reviews; an oracle would score 0.8957.
The result is about this budget, and the page says what would have to change.

The third capstone changes the question entirely: when there is no accuracy to report, what does a defensible result even look like? Capstone 3 - A Generative Model You Can Defend.

pch.quizTag pch.quizDefaultTitle

A bag-of-words linear model scored 0.8507 in 3 seconds while self-attention scored 0.7957 in 165 seconds. What is the correct conclusion?

pch.quizShowAnswer

B — At 6,000 reviews and 6 epochs a bag of words is the right model — sentiment is largely about which words appear, and there is not enough data to learn embeddings and a sequence function together — The published results that favour sequence models on IMDB use roughly four times this data. The finding is about the budget, and the page states what would change it.
On reviews of 400+ tokens the bag of words scored 0.8347 and the LSTM 0.7813. Why is that particularly damaging to the case for recurrence?

pch.quizShowAnswer

B — Long reviews are exactly where word order should matter most, so the sequence models losing by MORE there is the opposite of the prediction they are supposed to satisfy — Testing the specific claim a technique makes, rather than just overall accuracy, is what turns 'it lost' into 'it lost where it should have won'.
An oracle picking the better of two models per review would score 0.8957 against 0.8587 for the best single model. What does that justify?

pch.quizShowAnswer

B — An ensemble — the 0.0370 gap exists because 246 reviews are solved by exactly one of the two models, so their errors are not the same errors — The oracle is not directly achievable, but averaging the two models' probabilities captures part of it and is a two-line change.
43.3% of reviews exceed the 200-token window. How does that affect the comparison?

pch.quizShowAnswer

B — It limits every model equally — all of them read only the first 200 tokens of the longest reviews — but it must be reported, since it caps what any of them could achieve — Every model sees the same truncated input, so the ranking is fair; the truncation matters because it bounds the achievable accuracy and explains part of the drop on long reviews.
What should be reported when a ladder stops climbing?

pch.quizShowAnswer

B — The whole ladder including the negative gains, the budget every number is conditional on, and what you would change — the failed rungs are the finding — Omitting the rungs that lost turns a measurement into an advertisement, and it hides the information that would let someone else make a different budget decision.

🧪 Try It Yourself

pch.coffeeTagline

pch.coffeeCta

pch.feedbackHeading

pch.feedbackSubheading