The short-term memory problem

Every time data passes through a plain RNN cell, some information gets lost. By the time the network has read a long sequence, its hidden state barely remembers the earliest inputs — imagine trying to translate a long sentence but forgetting how it started by the time you reach the end. Plain (SimpleRNNSimpleRNN) cells are usually only good for patterns spanning around 10 steps.

To fix this, researchers introduced cells with an explicit long-term memory alongside the usual short-term state. These cells have been so successful that plain RNN cells are rarely used anymore. The most popular is the LSTM cell.

LSTM: Long Short-Term Memory

An LSTM cell looks like a regular cell from the outside, but its state is split into two vectors:

• h(t)h(t) — the short-term state (like a regular RNN’s hidden state)
• c(t)c(t) — the long-term state (the “cell” state — hence LSTM)

Internally, the network learns what to store, what to forget, and what to read from the long-term state, using three gates:

  flowchart LR
  subgraph LSTM["LSTM cell"]
    direction LR
    F["Forget gate f(t)"] -->|erase| C["Long-term state c(t-1) → c(t)"]
    I["Input gate i(t)"] -->|add| C
    G["Candidate memory g(t)"] --> I
    C --> O["Output gate o(t)"]
    O --> H["Short-term state / output h(t)"]
  end

• Forget gate f(t)f(t) — controls which parts of the long-term state get erased.
• Input gate i(t)i(t) — controls which parts of the new candidate memory g(t)g(t) get added to the long-term state.
• Output gate o(t)o(t) — controls which parts of the (updated) long-term state get read out as this step’s short-term state / output.

All three gates use a sigmoid activation, so their outputs range from 0 to 1 — think of them as “how much to let through,” multiplied element-wise against the state. A gate outputting near 0 closes; near 1 it opens.

In short: the input gate decides what’s worth remembering, the forget gate decides what to let go of, and the output gate decides what’s relevant right now. That’s what lets an LSTM carry information across far more time steps than a plain RNN cell.

In tf.keras, swapping a SimpleRNNSimpleRNN for an LSTMLSTM is a one-line change:

from tensorflow import keras
 
model = keras.models.Sequential([
    keras.layers.LSTM(20, return_sequences=True, input_shape=[None, 1]),
    keras.layers.LSTM(20, return_sequences=True),
    keras.layers.TimeDistributed(keras.layers.Dense(10)),
])
model.compile(loss="mse", optimizer="adam")
model.summary()

from tensorflow import keras
 
model = keras.models.Sequential([
    keras.layers.LSTM(20, return_sequences=True, input_shape=[None, 1]),
    keras.layers.LSTM(20, return_sequences=True),
    keras.layers.TimeDistributed(keras.layers.Dense(10)),
])
model.compile(loss="mse", optimizer="adam")
model.summary()

Keras also lets you build cells manually with keras.layers.RNNkeras.layers.RNN, e.g. keras.layers.RNN(keras.layers.LSTMCell(20), ...)keras.layers.RNN(keras.layers.LSTMCell(20), ...) — useful when you want a custom cell, but for standard use the LSTMLSTM layer is preferred since it uses an optimized implementation on GPU.

GRU: Gated Recurrent Unit

The Gated Recurrent Unit (GRU) cell is a simplified LSTM that tends to perform just as well, which is part of why it’s popular:

• the short-term and long-term states are merged into a single vector h(t)h(t)
• a single gate controller z(t)z(t) handles both forgetting and remembering: wherever it opens the input gate, it closes the forget gate, and vice versa — a memory slot gets erased right before it’s overwritten
• there’s no output gate — the full state is output at every time step
• a new reset gate r(t)r(t) controls how much of the previous state is shown to the main “candidate memory” layer

  flowchart LR
  subgraph GRU["GRU cell"]
    direction LR
    Z["Update gate z(t)"] -->|"mix old vs new"| H["State h(t)"]
    R["Reset gate r(t)"] -->|"forget part of h(t-1)"| G["Candidate g(t)"]
    G --> H
  end

Switching to a GRU is, again, just a layer swap:

from tensorflow import keras
 
model = keras.models.Sequential([
    keras.layers.GRU(20, return_sequences=True, input_shape=[None, 1]),
    keras.layers.GRU(20, return_sequences=True),
    keras.layers.TimeDistributed(keras.layers.Dense(10)),
])
model.compile(loss="mse", optimizer="adam")
model.summary()

from tensorflow import keras
 
model = keras.models.Sequential([
    keras.layers.GRU(20, return_sequences=True, input_shape=[None, 1]),
    keras.layers.GRU(20, return_sequences=True),
    keras.layers.TimeDistributed(keras.layers.Dense(10)),
])
model.compile(loss="mse", optimizer="adam")
model.summary()

LSTM vs. GRU vs. SimpleRNN

Both LSTM and GRU still struggle with very long sequences (100+ steps, like raw audio) — for those, later architectures mix in 1D convolutions or, ultimately, Transformers.

Mini-checkpoint

Why does an LSTM need both a forget gate and an input gate instead of just one?

• because forgetting old information and adding new information are separate decisions — you might want to keep everything and add something new, or clear space without adding anything yet.

Visualize it

Watch the three LSTM gates act on the long-term state c(t)c(t) as it flows left to right. The forget gate trims the incoming memory, the input gate blends in new information, and the output gate decides how much of the result becomes this step’s output:

🧪 Try It Yourself

Exercise 1 – Build an LSTM Layer

Exercise 2 – Swap in a GRU

Exercise 3 – Count Cell Parameters

Next

Continue to Time Series Forecasting with RNNs — put LSTM and GRU cells to work forecasting real sequences, and see how they compare against simple baselines.