What you’ll learn

• why gradients vanish (or explode) as they flow backward through a deep net
• Glorot (Xavier) and He initialization — and why the default matters
• nonsaturating activations: ReLU, Leaky ReLU, ELU, and SELU
• gradient clipping for when exploding gradients are the real problem

The problem

Backpropagation flows the error gradient from the output layer back to the input layer, and uses it to update every weight along the way. In a deep network, that gradient often gets smaller and smaller the further back it travels — by the time it reaches the first few layers there’s barely a signal left, so those layers stop learning. This is the vanishing gradients problem. The opposite can also happen: gradients grow larger and larger until the updates are enormous and training diverges — the exploding gradients problem (more common in RNNs).

Géron traces this back to a 2010 paper by Glorot and Bengio: with the once-popular combination of the logistic sigmoid activation and naive random initialization, the variance of each layer’s output keeps growing as you go deeper, until the activation saturates near 0 or 1 at the top layers — and a saturated sigmoid has a derivative close to zero, so almost no gradient survives the trip backward.

  flowchart LR
  L1["Layer 1
gradient ≈ 0.9"] --> L2["Layer 2
gradient ≈ 0.4"]
  L2 --> L3["Layer 3
gradient ≈ 0.1"]
  L3 --> L4["Layer 4
gradient ≈ 0.01"]
  L4 --> L5["Layer 5
gradient ≈ nearly 0"]

Glorot and He initialization

Glorot and Bengio’s fix: for the signal to flow properly in both directions (forward predictions, backward gradients), the variance of a layer’s outputs should roughly equal the variance of its inputs. That leads to an initialization scheme scaled by the layer’s fan-in and fan-out (its number of input and output connections):

Keras uses Glorot initialization with a uniform distribution by default. Switch to He initialization explicitly when you’re using ReLU-family activations:

import tensorflow as tf
 
# Default: Glorot uniform — fine for tanh/sigmoid/softmax
tf.keras.layers.Dense(10, activation="tanh")
 
# He initialization — pairs with ReLU and its variants
tf.keras.layers.Dense(10, activation="relu", kernel_initializer="he_normal")
 
# LeCun initialization — required for SELU's self-normalizing property
tf.keras.layers.Dense(10, activation="selu", kernel_initializer="lecun_normal")

import tensorflow as tf
 
# Default: Glorot uniform — fine for tanh/sigmoid/softmax
tf.keras.layers.Dense(10, activation="tanh")
 
# He initialization — pairs with ReLU and its variants
tf.keras.layers.Dense(10, activation="relu", kernel_initializer="he_normal")
 
# LeCun initialization — required for SELU's self-normalizing property
tf.keras.layers.Dense(10, activation="selu", kernel_initializer="lecun_normal")

Nonsaturating activation functions

The choice of activation matters just as much as initialization. ReLU (max(0, z)max(0, z)) doesn’t saturate for positive inputs and is cheap to compute, which is why it’s the default hidden-layer activation today — but it has its own “dying ReLU” problem: a neuron whose weighted input is always negative just outputs 0 forever, and gradient descent can’t revive it.

• Leaky ReLU — max(α·z, z)max(α·z, z) — gives negative inputs a small slope (αα, typically 0.010.01 or 0.20.2) instead of a hard zero, so neurons can’t fully die.
• ELU — smooth and negative for z < 0z < 0, which pushes the average output closer to zero and speeds up convergence, at the cost of being slower to compute.
• SELU — a scaled variant of ELU that, under the right conditions (LeCun normal init, all-Dense sequential architecture, standardized inputs), makes the network self-normalize: every layer’s output tends to keep a mean of 0 and a standard deviation of 1, all on its own.

Géron’s rule of thumb for general use: SELU > ELU > Leaky ReLU (and variants) > ReLU > tanh > logisticSELU > ELU > Leaky ReLU (and variants) > ReLU > tanh > logistic.

import tensorflow as tf
 
model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=[28, 28]),
    tf.keras.layers.Dense(300, kernel_initializer="he_normal", use_bias=False),
    tf.keras.layers.LeakyReLU(alpha=0.2),
    tf.keras.layers.Dense(100, activation="selu", kernel_initializer="lecun_normal"),
    tf.keras.layers.Dense(10, activation="softmax"),
])

import tensorflow as tf
 
model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=[28, 28]),
    tf.keras.layers.Dense(300, kernel_initializer="he_normal", use_bias=False),
    tf.keras.layers.LeakyReLU(alpha=0.2),
    tf.keras.layers.Dense(100, activation="selu", kernel_initializer="lecun_normal"),
    tf.keras.layers.Dense(10, activation="softmax"),
])

Gradient clipping

Batch Normalization (next page) usually handles unstable gradients well enough on its own, but gradient clipping is a simple, direct fix that’s especially common in recurrent networks: clip every gradient component to a fixed range before it’s applied.

import tensorflow as tf
 
# clip every gradient component to [-1.0, 1.0]
optimizer = tf.keras.optimizers.SGD(clipvalue=1.0)
 
# or clip the whole gradient vector by its L2 norm, preserving direction
optimizer = tf.keras.optimizers.SGD(clipnorm=1.0)
 
model = tf.keras.models.Sequential([tf.keras.layers.Dense(1, input_shape=(4,))])
model.compile(loss="mse", optimizer=optimizer)

import tensorflow as tf
 
# clip every gradient component to [-1.0, 1.0]
optimizer = tf.keras.optimizers.SGD(clipvalue=1.0)
 
# or clip the whole gradient vector by its L2 norm, preserving direction
optimizer = tf.keras.optimizers.SGD(clipnorm=1.0)
 
model = tf.keras.models.Sequential([tf.keras.layers.Dense(1, input_shape=(4,))])
model.compile(loss="mse", optimizer=optimizer)

clipvalueclipvalue can change the direction of the gradient vector (each component is clipped independently); clipnormclipnorm rescales the whole vector so its direction is preserved. Try both if you see exploding gradients in TensorBoard.

Where this fits in the bigger picture

François Chollet frames the same symptom — a training loss that refuses to move — as the first thing to debug in any deep learning project, and his first suspects are the learning rate and batch size (see Backpropagation and Optimizers, earlier in this phase). Vanishing or exploding gradients are the same failure one layer deeper: even with a perfectly sane learning rate, a bad init/activation combination can strangle the gradient signal before it ever reaches the optimizer step. If a model still won’t train after you’ve ruled out the optimizer’s hyperparameters, this page — not a fancier optimizer — is where to look next.

Visualize it

Watch what happens to the gradient signal as it’s multiplied backward through more and more layers — with a saturating activation it shrinks toward zero almost immediately, while a nonsaturating one keeps a usable signal much deeper:

🧪 Try It Yourself

Exercise 1 – He Initialization for ReLU

Exercise 2 – Add a Leaky ReLU Layer

Exercise 3 – Clip Gradients by Norm

Next

Continue to Batch Normalization — a technique that keeps every layer’s inputs well-scaled throughout training, not just at initialization.