Interpreting What Convnets Learn (Grad-CAM)

What you’ll learn

why convnets are much easier to interpret than people assume — their features are visual, so you can literally look at them
how to visualize a layer’s intermediate activations on a real image
how to visualize what a single filter is looking for, via gradient ascent
how Grad-CAM produces a heatmap showing exactly which pixels drove a prediction

Convnets aren’t really black boxes

Deep learning models are often called “black boxes” — hard to inspect, hard to explain. That reputation is mostly undeserved for convnets. Because their filters detect visual concepts (edges, textures, eyes, wheels), the representations they learn are unusually amenable to visualization: you can point a technique at a trained convnet and literally see what it’s paying attention to. That matters whenever a wrong prediction needs debugging, or whenever deep learning is meant to support (not replace) a human expert — like in medical imaging.

Three techniques cover most of what you need:

Intermediate activations — what does each layer’s output look like for a given input?
Filter visualization — what pattern is a specific filter maximally receptive to?
Grad-CAM heatmaps — which pixels of this image caused this prediction?

Visualizing intermediate activations

Every convolution and pooling layer in a trained model produces an activation — its output for a given input. Since each activation is a 3D volume (height, width, channels)(height, width, channels), and each channel is roughly independent, you can plot every channel of every layer as its own small grayscale image. Feed in a real photo of a cat and look at the third or fourth convolutional layer, and you’ll typically see feature maps that light up on edges, then textures, then increasingly abstract “cat-like” blobs the deeper you go — exactly the hierarchy Hubel and Wiesel described in the visual cortex.

peek_at_activations.py

from tensorflow import keras
 
model = keras.models.load_model("convnet_from_scratch_with_augmentation.keras")
 
# Build a model that outputs every Conv2D / MaxPooling2D activation,
# instead of just the final prediction
layer_outputs = [layer.output for layer in model.layers[1:] if "conv2d" in layer.name or "pooling" in layer.name]
activation_model = keras.Model(inputs=model.input, outputs=layer_outputs)
 
activations = activation_model.predict(img_tensor)   # img_tensor: shape (1, 180, 180, 3)
first_layer_activation = activations[0]
print(first_layer_activation.shape)   # e.g. (1, 178, 178, 32) -- 32 feature maps

peek_at_activations.py

from tensorflow import keras
 
model = keras.models.load_model("convnet_from_scratch_with_augmentation.keras")
 
# Build a model that outputs every Conv2D / MaxPooling2D activation,
# instead of just the final prediction
layer_outputs = [layer.output for layer in model.layers[1:] if "conv2d" in layer.name or "pooling" in layer.name]
activation_model = keras.Model(inputs=model.input, outputs=layer_outputs)
 
activations = activation_model.predict(img_tensor)   # img_tensor: shape (1, 180, 180, 3)
first_layer_activation = activations[0]
print(first_layer_activation.shape)   # e.g. (1, 178, 178, 32) -- 32 feature maps

Visualizing what a filter looks for

Instead of feeding in a real photo, you can ask a filter directly: “what image would make you fire the hardest?” Starting from random noise, you repeatedly nudge the pixels (via gradient ascent, not descent — you’re maximizing the filter’s activation instead of minimizing a loss) until you land on an image the filter responds to strongly. Do this for every filter in a layer of a pretrained Xception model, and you’ll see the same pattern every time: early layers respond to simple edges and colors, middle layers to textures (stripes, dots, waves), and deep layers to combinations that start looking like eyes, fur, or leaves.

filter_visualization_sketch.py

import tensorflow as tf
from tensorflow import keras
 
model = keras.applications.xception.Xception(weights="imagenet", include_top=False)
layer = model.get_layer(name="block3_sepconv1")
feature_extractor = keras.Model(inputs=model.input, outputs=layer.output)
 
def compute_loss(image, filter_index):
    activation = feature_extractor(image)
    return tf.reduce_mean(activation[:, 2:-2, 2:-2, filter_index])  # avoid border artifacts
 
@tf.function
def gradient_ascent_step(image, filter_index, learning_rate):
    with tf.GradientTape() as tape:
        tape.watch(image)
        loss = compute_loss(image, filter_index)
    grads = tape.gradient(loss, image)
    grads = tf.math.l2_normalize(grads)
    return image + learning_rate * grads

filter_visualization_sketch.py

import tensorflow as tf
from tensorflow import keras
 
model = keras.applications.xception.Xception(weights="imagenet", include_top=False)
layer = model.get_layer(name="block3_sepconv1")
feature_extractor = keras.Model(inputs=model.input, outputs=layer.output)
 
def compute_loss(image, filter_index):
    activation = feature_extractor(image)
    return tf.reduce_mean(activation[:, 2:-2, 2:-2, filter_index])  # avoid border artifacts
 
@tf.function
def gradient_ascent_step(image, filter_index, learning_rate):
    with tf.GradientTape() as tape:
        tape.watch(image)
        loss = compute_loss(image, filter_index)
    grads = tape.gradient(loss, image)
    grads = tf.math.l2_normalize(grads)
    return image + learning_rate * grads

Grad-CAM: which pixels caused this prediction?

The most practical of the three techniques answers a very concrete question: “the model says this is an African elephant — which part of the picture convinced it?” This family of techniques is called class activation map (CAM) visualization, and the specific version below is Grad-CAM (Selvaraju et al., 2017).

The idea: take the output feature map of the model’s last convolutional layer, and weigh every channel in it by “how important that channel is to the predicted class” — measured as the gradient of the class score with respect to that channel. Intuitively, you’re combining two spatial maps: “how strongly does each location activate each channel” weighted by “how much does each channel matter for this class,” giving you “how strongly does each location support this class.”

diagram Grad-CAM pipeline mermaid

  flowchart LR
  I["Input image"] --> B["Conv layers
(feature extractor)"]
  B --> L["Last conv layer output
(H, W, channels)"]
  L --> P["Classifier head"]
  P --> C["Predicted class score"]
  C -->|"gradient"| G["Per-channel importance
(pooled gradients)"]
  G --> W["Weight each channel
by its importance"]
  L --> W
  W --> H["Heatmap
(mean over channels, ReLU, normalize)"]
  H --> O["Overlay on original image"]

Gradients of the predicted class flow back only as far as the last convolutional layer, where they become per-channel importance weights.

grad_cam.py

import numpy as np
import tensorflow as tf
from tensorflow import keras
 
model = keras.applications.xception.Xception(weights="imagenet")
 
img_array = get_img_array(img_path, target_size=(299, 299))   # preprocessed, shape (1, 299, 299, 3)
 
last_conv_layer_name = "block14_sepconv2_act"
classifier_layer_names = ["avg_pool", "predictions"]
 
last_conv_layer = model.get_layer(last_conv_layer_name)
last_conv_layer_model = keras.Model(model.inputs, last_conv_layer.output)
 
classifier_input = keras.Input(shape=last_conv_layer.output.shape[1:])
x = classifier_input
for layer_name in classifier_layer_names:
    x = model.get_layer(layer_name)(x)
classifier_model = keras.Model(classifier_input, x)
 
# Gradient of the top predicted class w.r.t. the last conv layer's output
with tf.GradientTape() as tape:
    last_conv_layer_output = last_conv_layer_model(img_array)
    tape.watch(last_conv_layer_output)
    preds = classifier_model(last_conv_layer_output)
    top_pred_index = tf.argmax(preds[0])
    top_class_channel = preds[:, top_pred_index]
 
grads = tape.gradient(top_class_channel, last_conv_layer_output)
 
# Pool the gradients into one "importance" number per channel
pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2)).numpy()
last_conv_layer_output = last_conv_layer_output.numpy()[0]
for i in range(pooled_grads.shape[-1]):
    last_conv_layer_output[:, :, i] *= pooled_grads[i]
heatmap = np.mean(last_conv_layer_output, axis=-1)
 
# Keep only positive contributions, then scale to [0, 1]
heatmap = np.maximum(heatmap, 0)
heatmap /= np.max(heatmap)

grad_cam.py

import numpy as np
import tensorflow as tf
from tensorflow import keras
 
model = keras.applications.xception.Xception(weights="imagenet")
 
img_array = get_img_array(img_path, target_size=(299, 299))   # preprocessed, shape (1, 299, 299, 3)
 
last_conv_layer_name = "block14_sepconv2_act"
classifier_layer_names = ["avg_pool", "predictions"]
 
last_conv_layer = model.get_layer(last_conv_layer_name)
last_conv_layer_model = keras.Model(model.inputs, last_conv_layer.output)
 
classifier_input = keras.Input(shape=last_conv_layer.output.shape[1:])
x = classifier_input
for layer_name in classifier_layer_names:
    x = model.get_layer(layer_name)(x)
classifier_model = keras.Model(classifier_input, x)
 
# Gradient of the top predicted class w.r.t. the last conv layer's output
with tf.GradientTape() as tape:
    last_conv_layer_output = last_conv_layer_model(img_array)
    tape.watch(last_conv_layer_output)
    preds = classifier_model(last_conv_layer_output)
    top_pred_index = tf.argmax(preds[0])
    top_class_channel = preds[:, top_pred_index]
 
grads = tape.gradient(top_class_channel, last_conv_layer_output)
 
# Pool the gradients into one "importance" number per channel
pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2)).numpy()
last_conv_layer_output = last_conv_layer_output.numpy()[0]
for i in range(pooled_grads.shape[-1]):
    last_conv_layer_output[:, :, i] *= pooled_grads[i]
heatmap = np.mean(last_conv_layer_output, axis=-1)
 
# Keep only positive contributions, then scale to [0, 1]
heatmap = np.maximum(heatmap, 0)
heatmap /= np.max(heatmap)

Feed in a photo of two African elephants and this produces a heatmap that lights up almost entirely around the elephants’ heads and ears — the exact regions a person would point to and say “that’s clearly an elephant.”

Visualize it

The heatmap below is a toy Grad-CAM: a simple shape sits in a noisy background, and the “importance” heatmap glows brightest exactly over the shape the (fake) classifier is keying on. The heatmap continuously pulses in intensity, fading down and re-focusing on a new random shape every few seconds — the same “re-run Grad-CAM, see where it looks” loop you’d get pointing this technique at a real convnet. Click to jump to a new shape right away:

sketch A Grad-CAM heatmap over an image grid p5.js

The jet-colored heatmap pulses and periodically re-focuses on a new random shape -- brightest right over the object, fading toward the edges and background. Click to reroll immediately.

Mini-checkpoint

Why does Grad-CAM use the gradient of the class score, and not just the raw activations, to weigh each channel?

Raw activations only tell you “how strongly did this channel fire here” — not whether firing strongly actually helped or hurt the predicted class. The gradient tells you exactly that: how sensitive the class score is to each channel, so channels the model actually relies on get weighted more heavily than channels that just happened to activate.

🧪 Try It Yourself

These exercises reproduce Grad-CAM’s math offline in NumPy — the same three steps used inside grad_cam.pygrad_cam.py above, on tiny arrays so they run instantly.

Exercise 1 – Pool Gradients Into Channel Importance

Exercise 2 – Weight the Feature Map by Channel Importance

Exercise 3 – Clean Up the Heatmap (ReLU + Normalize)

Continue to Object Detection (Bounding Boxes and YOLO) — go from “what is in this image” to “what’s in it, and exactly where,” by predicting bounding boxes and cleaning them up with non-max suppression.

If this helped you, consider buying me a coffee ☕

Buy me a coffee