Padding, Masking and Variable-Length Sequences

Real sequences have different lengths. Tensors do not. Every sequence model therefore starts with a compromise — pad everything to one length — and that compromise introduces a bug you cannot see in the loss curve, only in the score.

Here is the headline, measured on 5,000 IMDB reviews with an identical SimpleRNN(32), five epochs, four setups:

Padding	mask_zero	Best validation accuracy
pre	False	0.5856
pre	True	0.5868
post	False	0.5036
post	True	0.5272

Three of those runs learn something. One of them scores chance on a balanced binary task, and nothing in the code looks wrong.

What you’ll learn

• What pad_sequences does, and why maxlen is a two-sided trade: at 50 tokens you keep 0.2045 of the text; at 800 tokens 0.6987 of your batch is padding.
• Why post-padding without a mask destroys a recurrent model but pre-padding survives.
• What mask_zero=True actually produces — a boolean tensor travelling alongside the data, not zeroed values.
• Which layers respect a mask and which silently ignore it.
• Why an unmasked GlobalAveragePooling1D divides every review by the wrong number: the mean weight on real tokens was 0.8098, and the shortest review got 0.0800.
• Bucketing and ragged tensors, and when the padding is cheap enough not to care.

pad_sequences and the cost of maxlen

IMDB reviews, 5,000 of them: min 16 tokens, median 182, mean 243.8, p95 633, max 1,851. One tensor has to hold all of them.

The only two decisions that matter

x = keras.preprocessing.sequence.pad_sequences(
    sequences,
    maxlen=200,        # every row becomes exactly this long
    padding="pre",     # where the zeros go:      "pre" = front, "post" = back
    truncating="pre",  # which end is cut off if the row is too long
)

maxlen	Share of the batch that is padding	Share of real tokens kept	Rows truncated
50	0.0030	0.2045	0.9802
100	0.0332	0.3965	0.8944
200	0.1902	0.6642	0.4386
400	0.4642	0.8790	0.1506
800	0.6987	0.9887	0.0206

figure 5,000 IMDB reviews: the length distribution and what each maxlen costs matplotlib

The two curves cross somewhere around 200 tokens, which is why that value appears in every IMDB tutorial. At 800 the model sees 98.9% of the text and spends 69.9% of its compute on zeros; at 50 it wastes nothing and reads a fifth of each review. There is no setting that is cheap and complete — the distribution has a 1,851-token tail and a 16-token floor.

Padding is not free even when it is harmless: at maxlen=800 a recurrent layer runs 800 timesteps sequentially, of which 559 carry no information.

Why post-padding breaks a recurrent layer

A recurrent layer’s output is its last hidden state. That is the whole explanation.

diagram Diagram mermaid

  flowchart LR
  subgraph pre["pre-padding: 0 0 0 great film"]
    A["h after zeros"] --> B["h after 'great'"] --> C["h after 'film'
= output"]
  end
  subgraph post["post-padding: great film 0 0 0"]
    D["h after 'great'"] --> E["h after 'film'"] --> F["h after zeros
= output"]
  end

With pre-padding the padding comes first, and whatever the state accumulated over those zeros is then overwritten by the actual review. The last step is a real word, so the output is about the review.

With post-padding the last step is padding. For a review 68 tokens long inside a 200-step tensor, the network’s answer is the state after 132 consecutive padding steps — and every one of those steps multiplies the state by the recurrent weights again. Whatever the review contributed decays away, and the measured result is 0.5036: chance.

figure The same SimpleRNN(32), four ways of handling the same data matplotlib

The post-padded unmasked run does not diverge, crash or warn — it converges smoothly to chance. Masking recovers about half the damage (0.5036 to 0.5272) but not all of it: the two post-padded runs also see less of each review, because truncating='pre' keeps a different part of the text than the pre-padded runs do. The lesson is not 'masking fixes post-padding', it is 'pre-padding makes the question moot for recurrent layers'.

The default is the safe one, and that is why this bug hides

pad_sequences defaults to padding="pre". Most tutorials therefore work without ever mentioning masking — and the first time you write padding="post" because a transformer tutorial told you to, your accuracy quietly halves.

What mask_zero actually does

It does not zero the padded vectors. It attaches a boolean tensor to the output that downstream layers may consult:

python

embedding = keras.layers.Embedding(10000, 4, mask_zero=True)
sample = np.array([[0, 0, 7, 9],
                   [4, 5, 6, 8]])
embedded = embedding(sample)                   # (2, 4, 4) — real numbers everywhere
mask = embedding.compute_mask(sample).numpy()
# [[False False  True  True]
#  [ True  True  True  True]]

With mask_zero=False, compute_mask returns None — there is no mask to ignore, which is why the failure is silent rather than an error.

Measured on a freshly seeded layer: the padded position’s embedding vector has $\sum |v| = 0.0939$, not zero. Token id 0 is an ordinary row of the embedding matrix and it gets ordinary gradients like any other. The mask is metadata; a layer that ignores it sees those numbers as data.

Layer	Respects a mask?
SimpleRNN, LSTM, GRU	yes — stops updating the state at masked steps and repeats the last real state
Bidirectional	yes, in both directions
GlobalAveragePooling1D	yes — since TF 2.x it divides by the unmasked count
GlobalMaxPooling1D	no in older versions; check your version
Dense, Conv1D	no — they process every position identically
Flatten	no, and it destroys the mask
MultiHeadAttention	only if you pass attention_mask yourself

The two dangerous rows are Conv1D and Dense: they consume the padded vectors as though they were words, and they never complain.

Pooling: the dilution nobody notices

GlobalAveragePooling1D over an unmasked padded batch computes

$$\frac{1}{T}\sum_{t=1}^{T} v_t = \frac{L}{T}\cdot\underbrace{\frac{1}{L}\sum_{t \in \text{real}} v_t}_{\text{what you wanted}} + \frac{T - L}{T}\cdot v_{\text{pad}}$$

where $L$ is the true length and $T$ is maxlen. Every review is scaled by $L/T$ — a factor that varies per row and encodes nothing but length.

figure How much of an averaged review is actually the review matplotlib

The spike at 1.0 is the 43.9% of reviews long enough to be truncated — they have no padding at all. The tail reaching 0.2 is short reviews, and the shortest in this sample gets weight 0.0800: 92% of its pooled vector is the embedding of the padding token. Masking recovered +0.0088 (pre) and +0.0076 (post). Small, because the network can partly learn to compensate — and free, because the fix is one keyword.

Two honest observations about that measurement:

1. The gain is small. +0.0088 is not the difference between working and broken. A trained embedding can push the padding token towards zero and partly cancel the dilution itself, which is exactly what it does here.
2. It is still worth taking. One keyword, no cost, and it removes a length signal the model was free to abuse. On tasks where sequence length correlates with the label — support tickets, medical notes — an unmasked average lets the model cheat by reading the length instead of the content.

Also worth noting: the averaging model scored 0.8004, while the best SimpleRNN(32) on this page managed 0.5868. Averaging the embeddings and throwing away all order beat the recurrent layer by 0.21. That is not a masking result, it is a warning about SimpleRNN over 200 timesteps — the subject of the next page.

Alternatives to one big rectangle

Approach	How	When
Pad to a global maxlen	pad_sequences	default; simple, wasteful on skewed data
Bucketing by length	tf.data + bucket_by_sequence_length	long-tailed lengths; each batch pads to its own longest row
Ragged tensors	tf.ragged.constant, keras.layers.Input(ragged=True)	genuinely variable length, layers that support it
Truncate hard	small maxlen	when the signal is front-loaded — sentiment often is

Bucketing is the practical win, and Exercise 6 measures it: padding all 5,000 reviews to the longest one (1,851 tokens) makes 0.8683 of the batch padding; five length buckets bring that to 0.3207 — 0.5476 of the compute recovered with no change to the model and no text discarded.

Bucketing, in the shape you will actually use it

dataset = tf.data.Dataset.from_generator(...)          # variable-length rows
dataset = dataset.bucket_by_sequence_length(
    element_length_func=lambda x, y: tf.shape(x)[0],
    bucket_boundaries=[100, 200, 400, 800],
    bucket_batch_sizes=[64, 64, 32, 16, 8],            # one more than boundaries
)

sketch Padding, masking and the last hidden state p5.js Pause Restart

Toggle pre/post padding and the mask, and watch which step produces the output the classifier sees.

sketch The measured table, ranked p5.js Pause Restart

Click a column to rank every row by it. The bars are that column's values and the highest and lowest are computed from the numbers, not written in.

Pitfalls

• padding="post" with a recurrent layer and no mask. Measured 0.5036 — chance — with no error, no warning and a smooth loss curve.
• Assuming mask_zero=True zeroes the padded vectors. It does not: the measured padded embedding had $\sum|v| = 0.1477$. It attaches a boolean mask that layers may ignore.
• Putting Flatten or Dense after a masked embedding. The mask is dropped and every padded position becomes a feature.
• Reserving id 0 for a real token. mask_zero=True means id 0 is padding. If your vocabulary starts at 0, every occurrence of that word is masked away.
• Choosing maxlen from the maximum length. At 800 here, 69.9% of the batch is padding to accommodate a 1,851-token tail. Use a percentile — p95 was 633 — or bucket.
• Truncating the wrong end. truncating="pre" keeps the end of the text. For sentiment that is often right; for a news headline task it is exactly wrong.
• Comparing a padded model against an unpadded baseline. The padded model sees less text; that is a data difference, not an architecture result.

Recap

• pad_sequences(maxlen=..., padding=..., truncating=...) is the only place these decisions are made, and its defaults (pre, pre) are the safe ones for RNNs.
• On 5,000 IMDB reviews: maxlen=200 keeps 0.6642 of the tokens and spends 0.1902 of the batch on padding; maxlen=800 keeps 0.9887 and spends 0.6987.
• A recurrent layer outputs its last hidden state, so post-padding hands the classifier the state after the zeros: 0.5036 against 0.5856 for pre-padding.
• mask_zero=True attaches a boolean mask; RNNs, Bidirectional and GlobalAveragePooling1D honour it, while Dense, Conv1D and Flatten do not.
• An unmasked average weights the real tokens by $L/T$ — mean 0.8098, minimum 0.0800 in this sample — and masking bought +0.0088.
• Bucketing by length is the fix that scales: padding everything to 1,851 tokens wastes 0.8683 of the batch, five buckets waste 0.3207.

Next

SimpleRNN managed 0.5868 here while averaging the embeddings scored 0.8004. The gates that fix it — and the exact reason a plain recurrent state cannot hold information for 200 steps — are next: LSTM & GRU Networks.

pch.quizTag pch.quizDefaultTitle

1. An identical SimpleRNN scored 0.5856 with pre-padding and 0.5036 with post-padding. Why?

   A Post-padding corrupts the embedding matrix B A recurrent layer's output is its last hidden state — with post-padding that state is the one after all the zeros, so the review's contribution has decayed away C pad_sequences reverses the sequence when padding='post' D The loss function cannot handle trailing zeros
   pch.quizShowAnswer

   B — A recurrent layer's output is its last hidden state — with post-padding that state is the one after all the zeros, so the review's contribution has decayed away — 0.5036 on a balanced binary task is chance, and nothing in the code or the loss curve indicates a problem.

2. What does mask_zero=True actually do to the padded positions?

   A Sets their embedding vectors to zero B Nothing to the values — the measured padded vector still had sum|v| = 0.1477. It attaches a boolean mask tensor that downstream layers may consult C Removes those timesteps from the tensor D Replaces them with the mean embedding
   pch.quizShowAnswer

   B — Nothing to the values — the measured padded vector still had sum|v| = 0.1477. It attaches a boolean mask tensor that downstream layers may consult — Which is why a Dense or Conv1D layer after the embedding happily consumes padding as data.

3. On this corpus, maxlen=800 keeps 0.9887 of the real tokens. What does it cost?

   A Nothing — longer is always better B 0.6987 of every batch becomes padding, and a recurrent layer runs all 800 timesteps sequentially, so most of the compute processes zeros C The embedding matrix grows by 4x D Truncation of 20% of rows
   pch.quizShowAnswer

   B — 0.6987 of every batch becomes padding, and a recurrent layer runs all 800 timesteps sequentially, so most of the compute processes zeros — The 1,851-token maximum dictates that setting for the sake of a handful of rows; the p95 was 633 and bucketing avoids the choice entirely.

4. An unmasked GlobalAveragePooling1D over pre-padded reviews gave the real tokens a mean weight of 0.8098, and 0.0800 for the shortest review. What is the risk?

   A Numerical overflow in the sum B Every review is scaled by its own length/maxlen ratio, which injects a length signal the model can use instead of the content — and short reviews are mostly the padding embedding C The gradient becomes zero D The pooling output is no longer differentiable
   pch.quizShowAnswer

   B — Every review is scaled by its own length/maxlen ratio, which injects a length signal the model can use instead of the content — and short reviews are mostly the padding embedding — Masking bought only +0.0088 here because the trained embedding partly cancels the dilution itself, but the fix is one keyword and it removes the shortcut.

5. Which of these layers will silently treat padded positions as real data?

   A LSTM with mask_zero=True upstream B Conv1D and Dense — neither consults the mask, and Flatten destroys it C Bidirectional(LSTM) D GlobalAveragePooling1D in current TensorFlow
   pch.quizShowAnswer

   B — Conv1D and Dense — neither consults the mask, and Flatten destroys it — Recurrent layers, Bidirectional and modern GlobalAveragePooling1D honour the mask; convolutional and dense layers process every position identically.

🧪 Try It Yourself

pch.exerciseTag pch.exerciseTitle

pch.exerciseTag pch.exerciseTitle

pch.exerciseTag pch.exerciseTitle

pch.exerciseTag pch.exerciseTitle

pch.exerciseTag pch.exerciseTitle

pch.exerciseTag pch.exerciseTitle