What pooling layers do

Once you understand convolutional layers, pooling layers are easy: their goal is to subsample (shrink) the feature maps, which reduces the computational load, the memory usage, and the number of parameters — limiting the risk of overfitting along the way.

Just like a convolutional layer, each neuron in a pooling layer looks at a small receptive field in the previous layer. The difference is that a pooling neuron has no weights — it just aggregates its inputs with a fixed function such as maxmax or meanmean.

Max pooling vs. average pooling

• Max pooling keeps only the largest value in each receptive field and drops the rest. With a 2 × 2 kernel and stride 2, only the strongest of every 4 pixels survives.
• Average pooling keeps the mean instead of the max.

Max pooling is more common in practice: it keeps the strongest signal and throws away the noise, and it gives the network a bit of translation invariance — if the input shifts by a pixel or two, the max-pooled output often stays exactly the same. That invariance is great for classification (“is there a cat somewhere?”), but it’s actually undesirable for tasks like semantic segmentation, where a shift in the input should produce a matching shift in the output (this is called equivariance, not invariance).

The downside: max pooling is destructive. Even a tiny 2 × 2 kernel with stride 2 throws away 75% of the input values.

  flowchart LR
  A["Feature map
(H, W, C)"] --> P["Max pooling
2x2 kernel, stride 2"]
  P --> B["Smaller feature map
(H/2, W/2, C)"]

tf.keras implementation

import tensorflow as tf
 
max_pool = tf.keras.layers.MaxPool2D(pool_size=2)     # stride defaults to pool_size
avg_pool = tf.keras.layers.AvgPool2D(pool_size=2)
 
# Global average pooling: outputs one number per feature map, per instance.
# Often used as the very last layer before the Dense output in modern CNNs.
global_avg_pool = tf.keras.layers.GlobalAvgPool2D()

import tensorflow as tf
 
max_pool = tf.keras.layers.MaxPool2D(pool_size=2)     # stride defaults to pool_size
avg_pool = tf.keras.layers.AvgPool2D(pool_size=2)
 
# Global average pooling: outputs one number per feature map, per instance.
# Often used as the very last layer before the Dense output in modern CNNs.
global_avg_pool = tf.keras.layers.GlobalAvgPool2D()

A typical CNN stack

A classic CNN architecture repeats the same pattern: a few convolutional layers (each followed by ReLU), then a pooling layer to shrink the image, then a few more convolutional layers, then another pooling layer, and so on. As the image gets smaller, it typically gets deeper — more feature maps — since a pooling layer halving the spatial size lets you afford to double the number of filters without an explosion in parameters or compute. At the top, a regular feedforward network turns the final feature maps into a prediction.

Here is a simple CNN for Fashion MNIST, straight out of the book:

import tensorflow as tf
 
model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(64, 7, activation="relu", padding="same", input_shape=[28, 28, 1]),
    tf.keras.layers.MaxPooling2D(2),
    tf.keras.layers.Conv2D(128, 3, activation="relu", padding="same"),
    tf.keras.layers.Conv2D(128, 3, activation="relu", padding="same"),
    tf.keras.layers.MaxPooling2D(2),
    tf.keras.layers.Conv2D(256, 3, activation="relu", padding="same"),
    tf.keras.layers.Conv2D(256, 3, activation="relu", padding="same"),
    tf.keras.layers.MaxPooling2D(2),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation="relu"),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.Dense(64, activation="relu"),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.Dense(10, activation="softmax"),
])
model.compile(loss="sparse_categorical_crossentropy", optimizer="adam", metrics=["accuracy"])
model.summary()

import tensorflow as tf
 
model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(64, 7, activation="relu", padding="same", input_shape=[28, 28, 1]),
    tf.keras.layers.MaxPooling2D(2),
    tf.keras.layers.Conv2D(128, 3, activation="relu", padding="same"),
    tf.keras.layers.Conv2D(128, 3, activation="relu", padding="same"),
    tf.keras.layers.MaxPooling2D(2),
    tf.keras.layers.Conv2D(256, 3, activation="relu", padding="same"),
    tf.keras.layers.Conv2D(256, 3, activation="relu", padding="same"),
    tf.keras.layers.MaxPooling2D(2),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation="relu"),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.Dense(64, activation="relu"),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.Dense(10, activation="softmax"),
])
model.compile(loss="sparse_categorical_crossentropy", optimizer="adam", metrics=["accuracy"])
model.summary()

Notice how the number of filters grows (64 → 128 → 256) each time a pooling layer halves the height and width — this CNN reaches over 92% accuracy on Fashion MNIST, clearly better than a plain dense network on the same task.

Memory requirements

Convolutional layers can be very memory-hungry, especially during training — backpropagation needs to keep every intermediate feature map around for the reverse pass. Consider a convolutional layer with 5 × 5 filters, outputting 200 feature maps of size 150 × 100, stride 1, “same” padding, fed a 150 × 100 RGB image:

• Parameters: (5 × 5 × 3 + 1) × 200 = 15,200(5 × 5 × 3 + 1) × 200 = 15,200 — small compared to a DenseDense layer (which would need 675 million parameters for the same input and output size).
• Compute per instance: each of the 200 feature maps has 150 × 100 neurons, each computing a weighted sum of 5 × 5 × 3 = 75 inputs — about 225 million float multiplications.
• RAM for the output: 200 × 150 × 100 × 32 bits ≈ 12 MB, for one instance. A batch of 100 instances needs 1.2 GB just for this one layer’s output.

If training crashes with an out-of-memory error, try: a smaller mini-batch size, a larger stride, removing a layer, 16-bit floats instead of 32-bit, or distributing the model across multiple devices.

Modern twist: residual connections around pooling blocks

Stacking convolution and pooling layers works well up to a point, but very deep stacks run into a different problem: backpropagation is a bit like the game of telephone — an error signal gets passed backward through function after function, and each one introduces a little noise. Stack enough layers and the gradient reaching the earliest layers can get overwhelmed by that noise entirely, so the network stops learning. This is the vanishing gradient problem.

The fix, popularized by ResNet, is to add the input of a block back to its output — a residual (skip) connection — so gradients have a noise-free shortcut all the way back, no matter how deep the main path grows. When a block includes a MaxPooling2DMaxPooling2D layer, the residual needs its own small strided 1 × 11 × 1 convolution to shrink and reshape it so the addition still lines up:

import tensorflow as tf
 
def residual_block(x, filters, pooling=False):
    residual = x
    x = tf.keras.layers.Conv2D(filters, 3, activation="relu", padding="same")(x)
    x = tf.keras.layers.Conv2D(filters, 3, activation="relu", padding="same")(x)
    if pooling:
        x = tf.keras.layers.MaxPooling2D(2, padding="same")(x)
        # match the pooling layer's downsampling with a strided 1x1 conv
        residual = tf.keras.layers.Conv2D(filters, 1, strides=2)(residual)
    elif filters != residual.shape[-1]:
        # number of filters changed but no pooling -- just reshape channels
        residual = tf.keras.layers.Conv2D(filters, 1)(residual)
    return tf.keras.layers.add([x, residual])

import tensorflow as tf
 
def residual_block(x, filters, pooling=False):
    residual = x
    x = tf.keras.layers.Conv2D(filters, 3, activation="relu", padding="same")(x)
    x = tf.keras.layers.Conv2D(filters, 3, activation="relu", padding="same")(x)
    if pooling:
        x = tf.keras.layers.MaxPooling2D(2, padding="same")(x)
        # match the pooling layer's downsampling with a strided 1x1 conv
        residual = tf.keras.layers.Conv2D(filters, 1, strides=2)(residual)
    elif filters != residual.shape[-1]:
        # number of filters changed but no pooling -- just reshape channels
        residual = tf.keras.layers.Conv2D(filters, 1)(residual)
    return tf.keras.layers.add([x, residual])

Every architecture on the next page that goes very deep (ResNet, and anything built on top of it) relies on exactly this pattern: convolution + pooling for the “processing” work, and a skip connection so depth doesn’t come at the cost of trainability.

Visualize it

A 2 × 2 max-pooling kernel with stride 2 slides across the feature map without overlapping, and only the brightest (largest) pixel in each 2 × 2 block survives into the smaller output:

Mini-checkpoint

Why is max pooling used more often than average pooling in modern CNNs?

• it keeps only the strongest activations (a cleaner signal for later layers), gives slightly stronger translation invariance, and is cheaper to compute.

🧪 Try It Yourself

Exercise 1 – Add a Max Pooling Layer

Exercise 2 – Check the Downsampled Shape

Exercise 3 – Global Average Pooling

Next

Continue to Famous CNN Architectures (LeNet to ResNet) — see how these same convolution and pooling building blocks were arranged differently over the years to win the ImageNet challenge.