Why refine a plain RNN

A single LSTM or GRU layer is often just the starting point. Once you have a baseline that beats a common-sense heuristic (see the previous page), three knobs from Deep Learning with Python (Chollet) usually decide what to try next:

• training is unstable — loss barely moves, swings wildly, or hits NaNNaN → fight the unstable-gradients problem with recurrent layer normalization.
• the model is overfitting — training loss keeps dropping, validation loss doesn’t → add recurrent dropout.
• the model has plateaued but isn’t overfitting badly → stack more recurrent layers for extra capacity.
• the task cares about order but not which direction you read it in (most text tasks) → try a bidirectional wrapper.

  flowchart TD
  subgraph Stacked["Stacking recurrent layers"]
    direction TB
    S1["GRU(32, return_sequences=True)"] --> S2["GRU(32, return_sequences=True)"]
    S2 --> S3["Dense(1)"]
  end
  subgraph Bi["Bidirectional wrapper"]
    direction LR
    F["Forward LSTM: reads a, b, c, d, e"] --> M["Merge (concat)"]
    Bwd["Backward LSTM: reads e, d, c, b, a"] --> M
    M --> Out["Combined output"]
  end

Recurrent layer normalization

Before touching dropout at all, it’s worth checking for a different symptom: unstable gradients. An unrolled RNN is really a very deep feedforward network wearing a trench coat — the same weights get reused at every time step — so it can suffer the same vanishing/exploding gradients problem that affects any deep net. The tell is in the training curve itself, not the gap between train and validation loss: if loss barely moves, jumps around instead of settling down, or turns to NaNNaN, that’s unstable gradients, not overfitting.

Ordinary BatchNormalizationBatchNormalization doesn’t help much here. As Hands-On Machine Learning (Géron) explains, it can’t be applied between time steps — only between stacked recurrent layers — because it needs statistics computed across a batch, and those aren’t meaningful mid-sequence. Layer Normalization fixes this: instead of normalizing across the batch, it normalizes across the features of a single instance, so it behaves identically at training and inference time and can safely be applied at every time step, right after the linear combination of inputs and hidden state.

Keras doesn’t expose a layer_norm=Truelayer_norm=True flag on LSTMLSTM/GRUGRU, so using it takes a small custom cell wrapped in keras.layers.RNNkeras.layers.RNN — built on the plain SimpleRNNCellSimpleRNNCell:

from tensorflow import keras
 
class LNSimpleRNNCell(keras.layers.Layer):
    def __init__(self, units, activation="tanh", **kwargs):
        super().__init__(**kwargs)
        self.state_size = units
        self.output_size = units
        self.simple_rnn_cell = keras.layers.SimpleRNNCell(units, activation=None)
        self.layer_norm = keras.layers.LayerNormalization()
        self.activation = keras.activations.get(activation)
 
    def call(self, inputs, states):
        outputs, new_states = self.simple_rnn_cell(inputs, states)
        norm_outputs = self.activation(self.layer_norm(outputs))
        return norm_outputs, [norm_outputs]
 
model = keras.models.Sequential([
    keras.layers.RNN(LNSimpleRNNCell(32), return_sequences=True,
                      input_shape=[None, 14]),
])
model.compile(optimizer="rmsprop", loss="mse", metrics=["mae"])

from tensorflow import keras
 
class LNSimpleRNNCell(keras.layers.Layer):
    def __init__(self, units, activation="tanh", **kwargs):
        super().__init__(**kwargs)
        self.state_size = units
        self.output_size = units
        self.simple_rnn_cell = keras.layers.SimpleRNNCell(units, activation=None)
        self.layer_norm = keras.layers.LayerNormalization()
        self.activation = keras.activations.get(activation)
 
    def call(self, inputs, states):
        outputs, new_states = self.simple_rnn_cell(inputs, states)
        norm_outputs = self.activation(self.layer_norm(outputs))
        return norm_outputs, [norm_outputs]
 
model = keras.models.Sequential([
    keras.layers.RNN(LNSimpleRNNCell(32), return_sequences=True,
                      input_shape=[None, 14]),
])
model.compile(optimizer="rmsprop", loss="mse", metrics=["mae"])

The cell delegates the actual recurrence math to SimpleRNNCellSimpleRNNCell (with activation=Noneactivation=None, so the raw linear combination comes out untouched), applies LayerNormalizationLayerNormalization, and only then applies the activation function — the same order used in feedforward nets: linear step, then normalize, then activate. Wrapping any custom cell in keras.layers.RNN(...)keras.layers.RNN(...) like this hands you back an ordinary Keras layer that still supports return_sequencesreturn_sequences, stacking, and everything else on this page. A quicker, coarser lever for the same problem is gradient clipping — passing clipnorm=1.0clipnorm=1.0 (or clipvalue=...clipvalue=...) to the optimizer caps how large any single gradient update can be, regardless of the layer architecture.

Recurrent dropout

Regular dropout — zeroing out random units — hurts recurrent layers if you’re not careful. Yarin Gal’s research (built directly into Keras) found the fix: use the same dropout mask at every timestep, instead of a fresh random mask each step. A mask that changes every step disrupts the error signal that has to flow backward through time; a fixed mask still regularizes without breaking that signal.

Every recurrent layer in Keras exposes two separate arguments:

• dropoutdropout — drops a fraction of the input units at each timestep.
• recurrent_dropoutrecurrent_dropout — drops a fraction of the recurrent (state-to-state) units, using that same fixed-per-layer mask idea.

from tensorflow import keras
from tensorflow.keras import layers
 
inputs = keras.Input(shape=(120, 14))
x = layers.LSTM(32, recurrent_dropout=0.25)(inputs)
x = layers.Dropout(0.5)(x)          # regularize the Dense head too
outputs = layers.Dense(1)(x)
model = keras.Model(inputs, outputs)
model.compile(optimizer="rmsprop", loss="mse", metrics=["mae"])

from tensorflow import keras
from tensorflow.keras import layers
 
inputs = keras.Input(shape=(120, 14))
x = layers.LSTM(32, recurrent_dropout=0.25)(inputs)
x = layers.Dropout(0.5)(x)          # regularize the Dense head too
outputs = layers.Dense(1)(x)
model = keras.Model(inputs, outputs)
model.compile(optimizer="rmsprop", loss="mse", metrics=["mae"])

Two practical notes from the book:

• networks regularized with dropout need more epochs to converge — the Jena temperature model above is trained for 50 epochs instead of 10.
• because you’re relying on dropout for regularization instead of a tiny layer, you can afford more units (32 here vs. 16 in the undropped baseline) without immediately overfitting.

On the Jena temperature task, this drops validation MAE from 2.36 to about 2.27 degrees — a real, if modest, improvement.

Stacking recurrent layers

If dropout fixed overfitting but accuracy has plateaued, the model is probably under capacity. The classic fix is the same one used for deep RNNs earlier in this phase: stack more recurrent layers. Every layer that feeds another recurrent layer must return its full sequence, so every layer except the last needs return_sequences=Truereturn_sequences=True:

from tensorflow import keras
from tensorflow.keras import layers
 
inputs = keras.Input(shape=(120, 14))
x = layers.GRU(32, recurrent_dropout=0.5, return_sequences=True)(inputs)
x = layers.GRU(32, recurrent_dropout=0.5)(x)
x = layers.Dropout(0.5)(x)
outputs = layers.Dense(1)(x)
model = keras.Model(inputs, outputs)
model.compile(optimizer="rmsprop", loss="mse", metrics=["mae"])

from tensorflow import keras
from tensorflow.keras import layers
 
inputs = keras.Input(shape=(120, 14))
x = layers.GRU(32, recurrent_dropout=0.5, return_sequences=True)(inputs)
x = layers.GRU(32, recurrent_dropout=0.5)(x)
x = layers.Dropout(0.5)(x)
outputs = layers.Dense(1)(x)
model = keras.Model(inputs, outputs)
model.compile(optimizer="rmsprop", loss="mse", metrics=["mae"])

On the Jena task, stacking two dropout-regularized GRU layers reaches a test MAE of about 2.39 degrees — better than a single layer, but only a small gain. That’s normal: once a small recurrent model has already fixed both underfitting and overfitting, extra layers tend to show diminishing returns, and each added layer costs noticeably more compute.

Bidirectional RNNs

A regular RNN reads a sequence in one direction — oldest step first. But that direction was really just a choice. A BidirectionalBidirectional layer runs two independent copies of a recurrent layer — one reading the sequence forward, one reading it backward — and merges their outputs (concatenation, by default):

from tensorflow import keras
from tensorflow.keras import layers
 
inputs = keras.Input(shape=(120, 14))
x = layers.Bidirectional(layers.LSTM(16))(inputs)
outputs = layers.Dense(1)(x)
model = keras.Model(inputs, outputs)
model.compile(optimizer="rmsprop", loss="mse", metrics=["mae"])

from tensorflow import keras
from tensorflow.keras import layers
 
inputs = keras.Input(shape=(120, 14))
x = layers.Bidirectional(layers.LSTM(16))(inputs)
outputs = layers.Dense(1)(x)
model = keras.Model(inputs, outputs)
model.compile(optimizer="rmsprop", loss="mse", metrics=["mae"])

Bidirectional(LSTM(16))Bidirectional(LSTM(16)) doubles the output width — each direction produces a 16-unit vector, concatenated into 32 — which also roughly doubles the recurrent layer’s capacity and training cost.

When bidirectional helps — and when it doesn’t

This is the counter-intuitive part. On the Jena temperature task, bidirectional actually performs worse than a plain LSTM: the recent past matters far more than the distant past for tomorrow’s weather, so reading the sequence backward mostly adds noisy, less-predictive capacity — and makes overfitting start sooner. But on text, order matters yet which direction you read in usually doesn’t — you can read a sentence backward and still catch its meaning — so a bidirectional layer genuinely helps there, catching patterns a one-directional pass would miss. That’s why bidirectional LSTMs were once the default choice for NLP tasks, before Transformers took over.

Mini-checkpoint

Why did the bidirectional LSTM underperform on the Jena temperature-forecasting task, when it’s a common upgrade for text models?

• because the direction you read a sequence in isn’t arbitrary for weather: the most recent readings are the most predictive, so the backward-reading half of the layer adds mostly noise instead of a genuinely useful new angle on the data — unlike text, where reading order doesn’t change the meaning.

Visualize it

Watch two passes over the same 5-step sequence — one left-to-right (blue), one right-to-left (amber) — before they’re merged into a single combined output:

🧪 Try It Yourself

Exercise 1 – Add Recurrent Dropout

Exercise 2 – Stack Two GRU Layers

Exercise 3 – Wrap an LSTM as Bidirectional

Next

Continue to Text Preprocessing (Tokenization, Stemming, Lemmatization) — start Phase 5 by turning raw text into the numeric sequences an RNN (or Transformer) can actually process.