What you’ll learn

• what Batch Normalization actually computes, batch by batch
• why it needs different behavior at training time vs. prediction time
• how to add BatchNormalizationBatchNormalization layers in tf.kerastf.keras, before or after the activation
• the momentummomentum hyperparameter, and why BN adds “non-trainable” parameters

Why BN, if we already fixed initialization?

Good initialization (Glorot/He) and a nonsaturating activation reduce the danger of vanishing or exploding gradients at the start of training — but they don’t guarantee the problem won’t come back once training is underway and the weights have moved. Batch Normalization (Ioffe & Szegedy, 2015) solves this directly by adding a normalizing step inside the network itself, just before or after each hidden layer’s activation function.

For each mini-batch, BN:

1. computes the mean and standard deviation of every input, across the batch
2. zero-centers and normalizes each input using those statistics
3. scales and shifts the result with two learned parameters per input: γγ (scale) and ββ (shift)

  flowchart LR
  X["Layer input"] --> M["Mean / std
(per mini-batch)"]
  M --> N["Normalize:
zero mean, unit variance"]
  N --> S["Scale (γ) and shift (β)"]
  S --> Y["Output to next layer"]

If you add a BN layer as the very first layer of your model, you often don’t even need a separate StandardScalerStandardScaler step — BN will approximately standardize the inputs for you, one batch at a time.

Training time vs. prediction time

Batch statistics only make sense when you actually have a batch. At prediction time you might be scoring a single instance, so BN can’t compute a mean or standard deviation from “the batch.” Instead, during training it also maintains a moving average of each input’s mean and standard deviation, and uses those frozen, final statistics once training is over. That’s why every BatchNormalizationBatchNormalization layer tracks four parameter vectors:

• γγ (scale) and ββ (shift) — learned through ordinary backpropagation
• moving_meanmoving_mean and moving_variancemoving_variance — updated via exponential moving average, not touched by backpropagation, which is why Keras reports them as non-trainable parameters

Adding BatchNormalization in Keras

The most common pattern: one BatchNormalizationBatchNormalization layer right after every hidden layer (and optionally one as the very first layer):

import tensorflow as tf
 
model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=[28, 28]),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dense(300, activation="elu", kernel_initializer="he_normal"),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dense(100, activation="elu", kernel_initializer="he_normal"),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dense(10, activation="softmax"),
])
 
model.summary()

import tensorflow as tf
 
model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=[28, 28]),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dense(300, activation="elu", kernel_initializer="he_normal"),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dense(100, activation="elu", kernel_initializer="he_normal"),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dense(10, activation="softmax"),
])
 
model.summary()

There’s some debate about whether BN should go before or after the activation. To put it before, drop the activation from the DenseDense layer, add a separate ActivationActivation layer after the BatchNormalizationBatchNormalization, and — since BN already has its own shift parameter — drop the DenseDense layer’s bias with use_bias=Falseuse_bias=False:

import tensorflow as tf
 
model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=[28, 28]),
    tf.keras.layers.Dense(300, kernel_initializer="he_normal", use_bias=False),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Activation("elu"),
    tf.keras.layers.Dense(100, kernel_initializer="he_normal", use_bias=False),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Activation("elu"),
    tf.keras.layers.Dense(10, activation="softmax"),
])

import tensorflow as tf
 
model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=[28, 28]),
    tf.keras.layers.Dense(300, kernel_initializer="he_normal", use_bias=False),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Activation("elu"),
    tf.keras.layers.Dense(100, kernel_initializer="he_normal", use_bias=False),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Activation("elu"),
    tf.keras.layers.Dense(10, activation="softmax"),
])

Checking which parameters of the first BN layer are trainable confirms the split described above:

for var in model.layers[1].variables:
    print(var.name, var.trainable)

for var in model.layers[1].variables:
    print(var.name, var.trainable)

batch_normalization/gamma:0 True
batch_normalization/beta:0 True
batch_normalization/moving_mean:0 False
batch_normalization/moving_variance:0 False

batch_normalization/gamma:0 True
batch_normalization/beta:0 True
batch_normalization/moving_mean:0 False
batch_normalization/moving_variance:0 False

The momentummomentum hyperparameter

momentummomentum controls how quickly the moving averages adapt to new batches. Given a new batch statistic vv, BN updates the running average v̂v̂ with:

v̂ ← v̂ · momentum + v · (1 − momentum)v̂ ← v̂ · momentum + v · (1 − momentum)

Good values are typically close to 11 — 0.90.9, 0.990.99, or 0.9990.999 — with more 9s for larger datasets and smaller mini-batches:

import tensorflow as tf
 
bn_layer = tf.keras.layers.BatchNormalization(momentum=0.99)

import tensorflow as tf
 
bn_layer = tf.keras.layers.BatchNormalization(momentum=0.99)

Where this fits in the bigger workflow

In François Chollet’s universal workflow of machine learning, “try different architectures” is one of the standard moves once a model has proven it can overfit and you’ve moved on to regularizing and tuning it. Adding BatchNormalizationBatchNormalization layers is exactly that kind of architecture change: it doesn’t touch your loss function or your data, but it changes how easily the network trains, which indirectly affects how much regularization (Dropout, L1/L2, early stopping) you’ll actually need on top of it.

Visualize it

Watch one feature’s activations across a mini-batch as training drifts its mean and spread around — and notice that no matter how wild the raw distribution gets, the normalized version on the right always snaps back to a clean, centered shape with mean 0 and standard deviation 1:

Mini-checkpoint

Why can Batch Normalization let you use a larger learning rate than usual?

• Because it keeps each layer’s inputs consistently scaled, the network becomes far less sensitive to the exact weight initialization or to large updates — the original paper reported reaching the same accuracy with 14× fewer training steps.

🧪 Try It Yourself

Exercise 1 – Add BatchNormalization to a Model

Exercise 2 – Count Trainable BN Parameters

Exercise 3 – Set the BN Momentum

Next

Continue to Regularization & Dropout — with unstable gradients under control, the next risk for a deep net is simply memorizing the training set.