What you’ll learn

• the classic LeNet-5 architecture that introduced convolution + pooling
• AlexNet, the deep-learning architecture that won ImageNet 2012
• GoogLeNet’s inception modules, and why 1 × 1 convolutions matter
• VGGNet’s “just stack more 3 × 3 convs” philosophy
• ResNet’s skip connections, and why they make very deep nets trainable
• Xception’s depthwise separable convolutions
• SENet’s squeeze-and-excitation blocks, which recalibrate feature maps by channel

A good way to track CNN progress is the ILSVRC ImageNet competition: the top-five error rate for image classification fell from over 26% to under 2.3% in just six years, almost entirely through architecture changes.

  flowchart LR
  A["LeNet-5
(1998)"] --> B["AlexNet
(2012)"]
  B --> C["GoogLeNet
(2014)"]
  B --> D["VGGNet
(2014)"]
  C --> E["ResNet
(2015)"]
  D --> E
  E --> F["Xception
(2016)"]
  F --> G["SENet
(2017)"]

LeNet-5 (1998)

Created by Yann LeCun for handwritten digit recognition (MNIST), LeNet-5 is where convolutional and pooling layers were introduced together for the first time. It’s small by modern standards: a few convolution + average pooling pairs, ending in fully connected layers with tanhtanh activations. MNIST images (28 × 28) are zero-padded to 32 × 32 before being fed in.

AlexNet (2012)

AlexNet crushed the 2012 ImageNet competition — a 17% top-five error rate versus 26% for the runner-up. It’s similar to LeNet-5, “only much larger and deeper,” and it was the first architecture to stack convolutional layers directly on top of each other instead of always pairing conv → pool. Its key innovations:

• ReLU activation instead of tanhtanh/sigmoid
• Dropout (50%) on the two big fully connected layers, to fight overfitting
• Data augmentation — random shifts, horizontal flips, and lighting changes, to artificially grow the training set
• Local response normalization (LRN) — a competitive normalization step where strongly activated neurons suppress neighboring feature maps at the same position, encouraging different filters to specialize

GoogLeNet (2014)

GoogLeNet pushed the top-five error rate below 7%, using roughly 10 times fewer parameters than AlexNet (about 6 million vs. 60 million). The trick is the inception module: the input is copied and fed through four parallel branches (different kernel sizes capturing patterns at different scales), then all the outputs are concatenated along the depth dimension.

Inception modules use 1 × 11 × 1 convolutions as bottleneck layers: a 1 × 1 filter can’t capture any spatial pattern (it only looks at one pixel), but it can capture cross-channel patterns and — crucially — reduce the number of feature maps before the more expensive 3 × 3 and 5 × 5 convolutions run. This cuts computation and parameter count substantially. GoogLeNet stacks nine of these inception modules, and finishes with a global average pooling layer instead of several dense layers — dropping the remaining spatial information (which is fine, since by then the feature maps are just 7 × 7) while sharply cutting the parameter count.

VGGNet (2014)

VGGNet, the ILSVRC 2014 runner-up, took the opposite approach to GoogLeNet: extreme simplicity. It repeats “2 or 3 convolutional layers + a pooling layer” over and over — 16 or 19 convolutional layers total, depending on the variant — using only 3 × 3 filters, but many of them, followed by a dense network with 2 hidden layers.

ResNet (2015)

Kaiming He et al. won ILSVRC 2015 with a Residual Network (ResNet), reaching a top-five error rate under 3.6% using a network 152 layers deep. The key innovation is the skip connection (shortcut connection): the input to a layer is also added directly to the output of a layer higher up the stack.

If you’re training a network to model a target function h(x)h(x), adding a skip connection means the network is now forced to model the residual f(x) = h(x) − xf(x) = h(x) − x instead. This is a big deal at initialization: a freshly initialized network’s weights are close to zero, so without a skip connection it outputs values close to zero — but with a skip connection it outputs a copy of its input, i.e. it starts out modeling the identity function. If the true target function is reasonably close to identity (often the case), training converges dramatically faster, and gradients can flow across the whole network even before every layer has learned something useful.

A deep residual network is just a long stack of residual units (RUs), each with two 3 × 33 × 3 convolutions (with Batch Normalization and ReLU) and a skip connection. When the number of filters doubles (and spatial size halves) between blocks, the skip connection uses a 1 × 11 × 1 convolution with stride 2 to match shapes before the addition.

import tensorflow as tf
 
class ResidualUnit(tf.keras.layers.Layer):
    def __init__(self, filters, strides=1, activation="relu", **kwargs):
        super().__init__(**kwargs)
        self.activation = tf.keras.activations.get(activation)
        self.main_layers = [
            tf.keras.layers.Conv2D(filters, 3, strides=strides, padding="same", use_bias=False),
            tf.keras.layers.BatchNormalization(),
            self.activation,
            tf.keras.layers.Conv2D(filters, 3, strides=1, padding="same", use_bias=False),
            tf.keras.layers.BatchNormalization(),
        ]
        self.skip_layers = []
        if strides > 1:
            self.skip_layers = [
                tf.keras.layers.Conv2D(filters, 1, strides=strides, padding="same", use_bias=False),
                tf.keras.layers.BatchNormalization(),
            ]
 
    def call(self, inputs):
        Z = inputs
        for layer in self.main_layers:
            Z = layer(Z)
        skip_Z = inputs
        for layer in self.skip_layers:
            skip_Z = layer(skip_Z)
        return self.activation(Z + skip_Z)
 
 
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv2D(64, 7, strides=2, input_shape=[224, 224, 3],
                                  padding="same", use_bias=False))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Activation("relu"))
model.add(tf.keras.layers.MaxPool2D(pool_size=3, strides=2, padding="same"))
 
prev_filters = 64
for filters in [64] * 3 + [128] * 4 + [256] * 6 + [512] * 3:
    strides = 1 if filters == prev_filters else 2
    model.add(ResidualUnit(filters, strides=strides))
    prev_filters = filters
 
model.add(tf.keras.layers.GlobalAvgPool2D())
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(10, activation="softmax"))
model.summary()

import tensorflow as tf
 
class ResidualUnit(tf.keras.layers.Layer):
    def __init__(self, filters, strides=1, activation="relu", **kwargs):
        super().__init__(**kwargs)
        self.activation = tf.keras.activations.get(activation)
        self.main_layers = [
            tf.keras.layers.Conv2D(filters, 3, strides=strides, padding="same", use_bias=False),
            tf.keras.layers.BatchNormalization(),
            self.activation,
            tf.keras.layers.Conv2D(filters, 3, strides=1, padding="same", use_bias=False),
            tf.keras.layers.BatchNormalization(),
        ]
        self.skip_layers = []
        if strides > 1:
            self.skip_layers = [
                tf.keras.layers.Conv2D(filters, 1, strides=strides, padding="same", use_bias=False),
                tf.keras.layers.BatchNormalization(),
            ]
 
    def call(self, inputs):
        Z = inputs
        for layer in self.main_layers:
            Z = layer(Z)
        skip_Z = inputs
        for layer in self.skip_layers:
            skip_Z = layer(skip_Z)
        return self.activation(Z + skip_Z)
 
 
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv2D(64, 7, strides=2, input_shape=[224, 224, 3],
                                  padding="same", use_bias=False))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Activation("relu"))
model.add(tf.keras.layers.MaxPool2D(pool_size=3, strides=2, padding="same"))
 
prev_filters = 64
for filters in [64] * 3 + [128] * 4 + [256] * 6 + [512] * 3:
    strides = 1 if filters == prev_filters else 2
    model.add(ResidualUnit(filters, strides=strides))
    prev_filters = filters
 
model.add(tf.keras.layers.GlobalAvgPool2D())
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(10, activation="softmax"))
model.summary()

In fewer than 40 lines, that’s the architecture that won ILSVRC 2015 — a nice demonstration of how expressive the Keras API is.

A batch normalization detail worth memorizing

Notice use_bias=Falseuse_bias=False on every Conv2DConv2D in ResidualUnitResidualUnit above, immediately followed by a BatchNormalizationBatchNormalization layer. That’s not an accident. Because BatchNormalizationBatchNormalization re-centers its input on zero anyway, the convolution’s own bias term becomes redundant — dropping it makes the layer very slightly leaner for free. The recommended order is also Conv2DConv2D (no bias) → BatchNormalizationBatchNormalization → activation, i.e. apply the activation function after normalizing, not before:

import tensorflow as tf
 
# Recommended: normalize first, then activate
x = tf.keras.layers.Conv2D(32, 3, use_bias=False, padding="same")(x)
x = tf.keras.layers.BatchNormalization()(x)
x = tf.keras.layers.Activation("relu")(x)

import tensorflow as tf
 
# Recommended: normalize first, then activate
x = tf.keras.layers.Conv2D(32, 3, use_bias=False, padding="same")(x)
x = tf.keras.layers.BatchNormalization()(x)
x = tf.keras.layers.Activation("relu")(x)

The intuition: batch normalization centers your data around zero, and ReLU uses zero as its cutoff point for keeping or dropping activations — so normalizing before the activation gets the most out of that cutoff. Your model will still train if you order it the other way, but this ordering is the more common convention in modern architectures.

Xception (2016)

Xception (“Extreme Inception”) replaces inception modules with depthwise separable convolutions. A regular convolution captures spatial patterns (shapes) and cross-channel patterns (e.g. “mouth + nose + eyes = face”) at the same time. A depthwise separable convolution assumes these can be modeled separately: first, one spatial filter runs independently per input channel; then a 1 × 11 × 1 convolution mixes across channels. This uses fewer parameters, less memory, and less compute than a regular convolution — and in practice, often performs better.

The parameter savings are concrete, not just theoretical. Take a 3 x 33 x 3 convolution with 64 input channels and 64 output channels:

• Regular Conv2DConv2D: 3 * 3 * 64 * 64 = 36,8643 * 3 * 64 * 64 = 36,864 trainable parameters.
• Depthwise separable (SeparableConv2DSeparableConv2D): 3 * 3 * 64 + 64 * 64 = 4,6723 * 3 * 64 + 64 * 64 = 4,672 trainable parameters — under 13% as many.

That gap only grows as filters or kernel sizes get larger, which is exactly why Xception can match or beat much larger regular convnets while training on comparatively modest hardware.

SENet (2017): recalibrating feature maps

The ILSVRC 2017 winner pushed the top-five error rate down to an astonishing 2.25% — not with a brand-new backbone, but by bolting a small extra block onto an existing architecture (inception modules and residual units both got the treatment, producing SE-Inception and SE-ResNet). That extra piece is the SE block (Squeeze-and-Excitation), and its whole job is to recalibrate a layer’s feature maps: for each channel, learn a single number between 0 and 1 saying “how relevant is this feature map right now,” then multiply every feature map by its number.

The intuition: an SE block might learn that “mouth,” “nose,” and “eyes” feature maps tend to activate together. If it sees strong mouth and nose activations but only weak eye activations, that’s suspicious — so it boosts the eye feature map’s weight, helping resolve genuine ambiguity instead of just trusting whichever filters happened to fire loudest. It’s a form of attention, but purely along the channel dimension — an SE block never looks for spatial patterns, only “which of my channels matter.”

An SE block is only three layers deep:

1. Global average pooling — squeeze each feature map down to a single number (so 256 feature maps become a 256-length vector).
2. A small hidden DenseDense layer with ReLU — squeeze that vector down further, typically to 1/16th its size. This bottleneck forces the block to learn a compact, general summary of which features co-occur, rather than memorizing every combination.
3. A DenseDense output layer with sigmoid — expand back out to one number per feature map, each between 0 and 1: the recalibration weights.

import tensorflow as tf
 
def se_block(x, ratio=16):
    filters = x.shape[-1]
    se = tf.keras.layers.GlobalAveragePooling2D()(x)                        # squeeze
    se = tf.keras.layers.Dense(filters // ratio, activation="relu")(se)     # bottleneck
    se = tf.keras.layers.Dense(filters, activation="sigmoid")(se)           # excite
    se = tf.keras.layers.Reshape((1, 1, filters))(se)
    return tf.keras.layers.Multiply()([x, se])   # scale each feature map by its weight

import tensorflow as tf
 
def se_block(x, ratio=16):
    filters = x.shape[-1]
    se = tf.keras.layers.GlobalAveragePooling2D()(x)                        # squeeze
    se = tf.keras.layers.Dense(filters // ratio, activation="relu")(se)     # bottleneck
    se = tf.keras.layers.Dense(filters, activation="sigmoid")(se)           # excite
    se = tf.keras.layers.Reshape((1, 1, filters))(se)
    return tf.keras.layers.Multiply()([x, se])   # scale each feature map by its weight

Dropping a se_block()se_block() call at the end of every inception module or residual unit is usually all it takes — the recalibration weights are learned end-to-end alongside everything else, with no extra supervision needed.

Mini-checkpoint

Which architecture introduced skip connections, and what problem do they solve?

• ResNet. They let the network default to the identity function at initialization and let gradients flow across very deep stacks, which is what made 100+ layer networks trainable in the first place.

Visualize it

A residual unit’s skip connection carries the input xx straight to the addition, in parallel with the main path that has to pass through several convolutional layers — so the signal always has a fast lane through the network, no matter how deep the main path grows:

🧪 Try It Yourself

Exercise 1 – A Bottleneck 1x1 Convolution

Exercise 2 – Add a Skip Connection

Exercise 3 – Depthwise Separable Convolution

Next

Continue to Transfer Learning Using Pre-trained Models — instead of training a ResNet or Xception from scratch, download one already trained on ImageNet and adapt it to your own dataset.