Normalisation Beyond Batch: Layer, Group and Instance

Batch normalisation estimates its statistics from the batch, and that measurement showed it losing at batch size 4. Three alternatives fix that by reducing over different axes entirely. They differ in exactly one way — which values go into one mean — and everything else about their behaviour follows from that choice.

What you’ll learn

• All four normalisers implemented by hand and matched to Keras within 1.19e−06.
• The axes each reduces over, counted: on one tensor, 6, 8, 24 and 48 independent mean/variance pairs.
• What each actually centres — batch norm leaves per-sample means at 1.42e−01; layer norm leaves per-channel means at 1.86e+00.
• A batch-size sweep where LayerNormalization won at batches 4, 16 and 64 and batch norm won at 256.
• Why every method’s accuracy falls as the batch grows here, and why that is a confound rather than a result.
• The one-line relationship between instance norm and group norm.

One difference, four layers

Every one of these computes $\hat{x} = (x - \mu)/\sqrt{\sigma^2 + \epsilon}$ and then applies a learned scale and shift. The only question is which elements of the tensor $\mu$ and $\sigma$ are computed over. For a feature map of shape $(N, H, W, C)$:

Normaliser	Reduces over	One mean per	Depends on the batch?
BatchNormalization	$N, H, W$	channel	yes
LayerNormalization	$H, W, C$	sample	no
GroupNormalization(g)	$H, W, C/g$	sample × group	no
instance norm	$H, W$	sample × channel	no

figure The same tensor under four normalisers matplotlib

Each panel is the per-sample, per-channel mean after normalising an (8, 4, 4, 6) tensor whose six channels were deliberately given means from -2.0 to 18.4. Batch norm flattens the columns — it centres each channel across the batch. Layer norm flattens the rows — it centres each sample across its channels. Group norm does it in blocks of two channels, and instance norm centres every cell independently.

On an $(8, 4, 4, 6)$ tensor whose channels were given means from −2.02 to 18.45 and standard deviations from 0.19 to 8.34:

Normaliser	Mean/variance pairs	Values per estimate	max |hand-written − Keras|
batch	6	128	1.19e−06
layer	8	96	7.15e−07
group (3 groups)	24	32	4.77e−07
instance	48	16	9.09e−07

The two columns move in opposite directions, and that is the whole trade-off. More independent estimates means each one is computed from fewer values — batch norm has 6 estimates over 128 values each, instance norm has 48 over 16 each. Noisy estimates hurt; too few estimates means less normalising.

What each one actually centres

Measured after normalising, taking the maximum absolute mean over each axis:

Normaliser	Per-channel means	Per-sample means
batch	1.18e−06	1.42e−01
layer	1.86e+00	4.97e−08
group	8.71e−01	5.46e−08
instance	1.43e−07	1.29e−07

Read the two columns as a contradiction that is not one. Batch norm centres channels and leaves samples off-centre; layer norm does the exact reverse. Layer norm leaves a per-channel mean of 1.86 — it never promised to remove that, because it never looks across the batch. Only instance norm, which uses the most estimates, centres both.

If your channels are on wildly different scales and the batch is large, batch norm is measuring the thing you want. If each sample has its own scale — different lighting, different speaker volume, different sequence length — layer norm is.

All four, in Keras

keras.layers.BatchNormalization(momentum=0.9)
keras.layers.LayerNormalization()
keras.layers.GroupNormalization(groups=8)
keras.layers.GroupNormalization(groups=channels)   # this is instance norm

The last line is not a trick: instance normalisation is group normalisation with one channel per group. Keras ships no separate layer because it does not need one.

The statistic-count argument, before any training

figure 16x16 feature map, 64 channels, groups of 8 matplotlib

Only batch norm's line slopes: its estimate quality is a function of the batch size, while the other three are fixed by the feature-map geometry. At batch 1 batch norm averages 256 values — the same as instance norm — and by batch 128 it averages 32,768, more than any other method. That crossing is the entire practical argument.

Batch	Batch norm	Layer norm	Group norm (8)	Instance norm
1	256	16,384	2,048	256
8	2,048	16,384	2,048	256
32	8,192	16,384	2,048	256
128	32,768	16,384	2,048	256

Three things follow without training anything:

• Batch norm is the only one whose estimate quality depends on the batch. At batch 1 it has as little information as instance norm; at batch 128 it has twice layer norm’s.
• The others are decided by the feature map, not the batch — so they behave identically at batch 1 and batch 1,000, which is why every Transformer uses layer norm and why segmentation and detection models, which use tiny batches of large images, use group norm.
• Group norm is the tunable middle. groups=1 is layer norm, groups=channels is instance norm, and 8 or 32 is the usual compromise.

Measured: accuracy against batch size

Same convnet, same seed, 4,000 Fashion-MNIST rows, 6 epochs, only the normalisation layer changing:

figure Fashion-MNIST, 4,000 rows — which normaliser survives a small batch? matplotlib

LayerNormalization leads at batches 4, 16 and 64 (0.7980, 0.7860, 0.7410) and batch norm overtakes it at 256 (0.7090 against 0.6660), exactly where its estimates become the best-informed. Every line falls as the batch grows because the epoch count is fixed: batch 4 gets 6,000 optimiser steps and batch 256 gets 96. Compare within a column, not along a row.

Normaliser	Batch 4	Batch 16	Batch 64	Batch 256
none	0.7520	0.7010	0.6180	0.4440
batch	0.7590	0.7590	0.6890	0.7090
layer	0.7980	0.7860	0.7410	0.6660
group (8)	0.7510	0.7500	0.7030	0.6550

The confound first, because it dominates the table. Every column falls as the batch grows, and that is almost entirely because the epoch budget is fixed: batch 4 takes 6,000 optimiser steps and batch 256 takes 96. Nothing in this table says “large batches are bad” — it says “96 updates is not many”. The valid comparison is down each column, between normalisers at the same batch size.

With that said:

1. Layer norm won three of four batch sizes, by 0.039, 0.027 and 0.052. On a small convnet with a small batch it was simply the best choice.
2. Batch norm overtook it at 256 (0.7090 against 0.6660) — precisely where the statistic count crosses over. The theory and the measurement agree.
3. Batch norm’s advantage over nothing at all grows with batch size: +0.0070 at batch 4, +0.2650 at batch 256. At batch 4 it is barely worth the layer.
4. Group norm never won. It sat between the others at every size — which is what a compromise looks like, and why it is chosen for constraints rather than for accuracy.

The bug this page hit first

The first version of this comparison reported batch norm at 0.1740 at batch 64 and 0.1260 at batch 256 — collapsing as the batch grew. The cause was not the normaliser: at Keras’ default momentum=0.99, BN’s moving statistics need roughly 460 optimiser steps, and a larger batch means fewer steps. Batch 256 for 6 epochs is 96 steps, so the inference path was normalising with statistics that had barely moved from their initial values.

	Batch 64	Batch 256
momentum=0.99 (default)	0.1740	0.1260
momentum=0.9	0.6890	0.7090

This is the second time the same default produced a nonsense result in this phase — the architectures page hit it too. On any short run, or any run with a large batch, lower the BN momentum. Layer, group and instance norm have no such setting, because they keep no running statistics at all — which is a real operational advantage independent of accuracy.

diagram Diagram mermaid

  flowchart TD
  A["which normaliser?"] --> B{"batch size at least ~32
and stable?"}
  B -- yes --> C["BatchNormalization
lower the momentum on short runs"]
  B -- no --> D{"is each sample its own
scale? sequences, text"}
  D -- yes --> E["LayerNormalization
no batch dependence at all"]
  D -- "no: images, few per batch" --> F["GroupNormalization(8 or 32)"]
  G["style transfer, per-image
appearance"] --> H["instance norm
= GroupNormalization(groups=C)"]

sketch Pick the axes, watch the estimate count p5.js Pause Restart

A tensor of shape (N, H, W, C). Change the batch size and the group count and read how many values go into one mean for each normaliser.

sketch The measured table, ranked p5.js Pause Restart

Click a column to rank every row by it. The bars are that column's values and the highest and lowest are computed from the numbers, not written in.

Pitfalls

• Leaving BN at momentum=0.99 with a large batch. Fewer steps per epoch means the moving statistics never converge: 0.1260 against 0.7090 at batch 256.
• Expecting layer norm to centre channels. It leaves per-channel means at 1.86 by construction, because it never looks across the batch.
• Using batch norm at batch 1 or 2. It then averages as few values as instance norm while still carrying moving-statistics baggage.
• Reading a batch-size sweep at fixed epochs as a batch-size result. Batch 4 got 6,000 updates and batch 256 got 96.
• Reaching for group norm to improve accuracy. It never won a column here; it wins when the batch cannot be large.
• Looking for an InstanceNormalization layer in Keras. It is GroupNormalization(groups=channels).
• Mixing normalisers within a block. Each assumes it sees the un-normalised distribution; stacking two of them wastes parameters and confuses the scale.

Recap

• All four compute the same formula and differ only in which axes they reduce over, matched to Keras within 1.19e−06.
• More independent estimates means fewer values per estimate: batch 6×128, layer 8×96, group 24×32, instance 48×16 on the tensor measured.
• Batch norm centres channels and not samples (per-sample mean 1.42e−01); layer norm does the reverse (per-channel mean 1.86).
• Only batch norm’s estimate quality depends on the batch size — 256 values at batch 1 against 32,768 at batch 128.
• Measured on 4,000 rows: layer norm won at batches 4, 16 and 64; batch norm overtook it at 256, exactly where the statistic counts cross.
• Group norm sat between the others at every batch size — a compromise chosen for constraints, not accuracy.
• Instance norm is GroupNormalization(groups=channels).

Next

Back to the practical question every small-data vision project starts with — whether to train at all, or to start from weights someone else paid for: Transfer Learning Using Pre-trained Models.

pch.quizTag pch.quizDefaultTitle

1. After layer normalisation, the maximum per-channel mean is 1.86 rather than near zero. Is that a bug?

   A Yes — normalisation should centre everything B No — layer norm reduces over H, W and C within each sample, so it centres samples and never looks across the batch; per-channel means are not its business C Yes — the epsilon is too large D No, but only because the channels were on different scales
   pch.quizShowAnswer

   B — No — layer norm reduces over H, W and C within each sample, so it centres samples and never looks across the batch; per-channel means are not its business — Measured: layer norm leaves per-sample means at 4.97e-08 and per-channel means at 1.86. Batch norm is the exact reverse.

2. Your model uses BatchNormalization at batch 256 for 6 epochs on 4,000 rows and validates at 0.126 while training fine. What is wrong?

   A The batch is too large for the dataset B That is only 96 optimiser steps, and at the default momentum=0.99 BN's moving statistics need roughly 460 — the inference path is normalising with statistics that never converged C Batch norm does not work above batch 128 D The learning rate needs to scale with the batch size
   pch.quizShowAnswer

   B — That is only 96 optimiser steps, and at the default momentum=0.99 BN's moving statistics need roughly 460 — the inference path is normalising with statistics that never converged — Measured: 0.1260 at momentum=0.99 against 0.7090 at momentum=0.9, same everything else. Large batches make this worse, not better, because they mean fewer steps.

3. Why do Transformers use LayerNormalization rather than BatchNormalization?

   A Because it has fewer parameters B Because its statistics come from within each sample, so they are identical at batch 1 and batch 1,000 and do not depend on what else is in the batch — and it keeps no running statistics to converge C Because batch norm cannot handle sequences of different lengths at all D Because layer norm trains faster
   pch.quizShowAnswer

   B — Because its statistics come from within each sample, so they are identical at batch 1 and batch 1,000 and do not depend on what else is in the batch — and it keeps no running statistics to converge — Sequence models often have small or variable batches, and inference is frequently a single sequence. Layer norm is unaffected by both.

4. What is GroupNormalization(groups=channels) equivalent to?

   A Batch normalisation B Instance normalisation — one mean and variance per sample per channel, which is why Keras ships no separate layer for it C Layer normalisation D No normalisation at all
   pch.quizShowAnswer

   B — Instance normalisation — one mean and variance per sample per channel, which is why Keras ships no separate layer for it — And groups=1 is layer normalisation. Group norm spans the whole range between the two extremes.

5. In the measured sweep every normaliser's accuracy fell as the batch size grew. What does that show?

   A Large batches always generalise worse B Mostly that the epoch budget was fixed, so batch 4 got 6,000 optimiser steps and batch 256 got 96 — the valid comparison is between normalisers at the same batch size, not across batch sizes C That normalisation stops working above batch 64 D That the dataset is too small for large batches
   pch.quizShowAnswer

   B — Mostly that the epoch budget was fixed, so batch 4 got 6,000 optimiser steps and batch 256 got 96 — the valid comparison is between normalisers at the same batch size, not across batch sizes — Reading along a row conflates the normaliser with the update count. Reading down a column is the controlled comparison.

🧪 Try It Yourself

pch.exerciseTag pch.exerciseTitle

pch.exerciseTag pch.exerciseTitle

pch.exerciseTag pch.exerciseTitle

pch.exerciseTag pch.exerciseTitle

pch.exerciseTag pch.exerciseTitle