Skip to content
Python Central Hub

Neural Style Transfer

What you’ll learn

  • what style transfer means: keep a target image’s content, but repaint it in a reference image’s style
  • the content loss — an L2 distance between high-level convnet activations that keeps the generated image’s macrostructure intact
  • the style loss, built from the Gram matrix of a layer’s activations — a map of which features tend to fire together, capturing texture rather than layout
  • the total-variation loss that keeps the generated image spatially smooth instead of speckled
  • how these three losses combine into one objective that you optimize directly on the pixels of the output image

Repainting a photo in someone else’s style

Neural style transfer takes two images — a content image (say, a photo of a city) and a style reference (say, Van Gogh’s Starry Night) — and produces a third image that keeps the buildings and streets of the first while adopting the swirling brushstrokes and colors of the second.

diagram Combining content and style into one generated image mermaid
Two source images are scored by the same losses against a generated image, which is optimized until it satisfies both.

The one idea underneath every deep learning technique applies here too: define a loss for what you want, then minimize it. What’s unusual is what gets updated — not a model’s weights, but the pixels of the generated image itself. You start the generated image as a copy of the content image, then repeatedly nudge its pixels downhill against a loss that rewards content-similarity to one image and style-similarity to another.

text
loss = distance(content(original) , content(generated))
     + distance(style(reference)  , style(generated))
text
loss = distance(content(original) , content(generated))
     + distance(style(reference)  , style(generated))

The whole trick is finding functions content(...)content(...) and style(...)style(...) that actually capture what those words mean — and a pretrained convnet (VGG19, in the original paper) turns out to define both beautifully.

Content loss: match the high-level activations

Early convnet layers encode local detail (edges, small textures); later layers encode global, abstract structure — what is in the image rather than exactly where every pixel sits. So the content loss is simply the squared difference between one upper layer’s activations for the content image and for the generated image: match those, and the generated image will “look like the same scene” to the network, even if brushstroke-level detail differs.

Content loss (Keras / TensorFlow)
import tensorflow as tf
 
def content_loss(base_image_features, combination_image_features):
    return tf.reduce_sum(tf.square(combination_image_features - base_image_features))
Content loss (Keras / TensorFlow)
import tensorflow as tf
 
def content_loss(base_image_features, combination_image_features):
    return tf.reduce_sum(tf.square(combination_image_features - base_image_features))

Style loss: match the Gram matrix, not the pixels

Style is trickier — it isn’t “these exact activations,” it’s “these textures, at every scale.” Gatys et al.’s insight was to look at the Gram matrix of a layer’s feature maps: flatten each channel into a vector and take the inner product of every channel with every other channel. The result is a channels x channelschannels x channels table of how correlated each pair of feature detectors is across the whole image — a statistical fingerprint of texture that has thrown away spatial layout entirely.

Style loss via the Gram matrix (Keras / TensorFlow)
import tensorflow as tf
 
def gram_matrix(x):
    x = tf.transpose(x, (2, 0, 1))                      # (channels, height, width)
    features = tf.reshape(x, (tf.shape(x)[0], -1))       # (channels, height*width)
    return tf.matmul(features, tf.transpose(features))   # (channels, channels)
 
def style_loss(style_features, combination_features, img_height, img_width):
    S = gram_matrix(style_features)
    C = gram_matrix(combination_features)
    channels = 3
    size = img_height * img_width
    return tf.reduce_sum(tf.square(S - C)) / (4.0 * (channels ** 2) * (size ** 2))
Style loss via the Gram matrix (Keras / TensorFlow)
import tensorflow as tf
 
def gram_matrix(x):
    x = tf.transpose(x, (2, 0, 1))                      # (channels, height, width)
    features = tf.reshape(x, (tf.shape(x)[0], -1))       # (channels, height*width)
    return tf.matmul(features, tf.transpose(features))   # (channels, channels)
 
def style_loss(style_features, combination_features, img_height, img_width):
    S = gram_matrix(style_features)
    C = gram_matrix(combination_features)
    channels = 3
    size = img_height * img_width
    return tf.reduce_sum(tf.square(S - C)) / (4.0 * (channels ** 2) * (size ** 2))

Match the Gram matrices of the style image and the generated image at several layers — low ones for fine texture, high ones for larger patterns — and the generated image ends up sharing the reference’s textures at every scale, without ever being forced to copy its actual layout.

Total-variation loss: keep it smooth

A third, smaller term regularizes the generated image itself: it penalizes large differences between neighboring pixels, which discourages the noisy, speckled artifacts that pure gradient-based pixel optimization tends to produce.

Total-variation loss (Keras / TensorFlow)
import tensorflow as tf
 
def total_variation_loss(x, img_height, img_width):
    a = tf.square(x[:, : img_height - 1, : img_width - 1, :] - x[:, 1:, : img_width - 1, :])
    b = tf.square(x[:, : img_height - 1, : img_width - 1, :] - x[:, : img_height - 1, 1:, :])
    return tf.reduce_sum(tf.pow(a + b, 1.25))
Total-variation loss (Keras / TensorFlow)
import tensorflow as tf
 
def total_variation_loss(x, img_height, img_width):
    a = tf.square(x[:, : img_height - 1, : img_width - 1, :] - x[:, 1:, : img_width - 1, :])
    b = tf.square(x[:, : img_height - 1, : img_width - 1, :] - x[:, : img_height - 1, 1:, :])
    return tf.reduce_sum(tf.pow(a + b, 1.25))

Verifying the loss math with plain NumPy

All three losses are simple enough to check with toy “feature maps” — small arrays standing in for a conv layer’s output — with no TensorFlow required:

Content, style, and total loss on toy feature maps (verified with NumPy)
import numpy as np
 
np.random.seed(1)
channels, spatial = 3, 4   # toy: 3 "feature channels", 4 "spatial locations"
 
# --- content loss: how far is the generated image from the content image? ---
content_feat = np.random.uniform(0, 1, size=(channels, spatial))
combo_feat = content_feat + np.random.normal(0, 0.1, size=(channels, spatial))
 
def content_loss(base, combination):
    return np.sum((combination - base) ** 2)
 
cl = content_loss(content_feat, combo_feat)
print("content loss:", round(cl, 4))
 
# --- style loss: how far is the generated image's Gram matrix from the style's? ---
def gram_matrix(x):
    return x @ x.T   # channels x channels correlation table
 
style_feat = np.random.uniform(0, 1, size=(channels, spatial))
combo_feat2 = style_feat + np.random.normal(0, 0.2, size=(channels, spatial))
 
def style_loss(style, combination, size):
    S, C = gram_matrix(style), gram_matrix(combination)
    return np.sum((S - C) ** 2) / (4.0 * (style.shape[0] ** 2) * (size ** 2))
 
sl = style_loss(style_feat, combo_feat2, spatial)
print("style loss:", round(sl, 6))
 
# --- combine with weights, exactly like the book's compute_loss ---
content_weight, style_weight = 0.25, 1.0
total = content_weight * cl + style_weight * sl
print("total loss:", round(total, 4))
Content, style, and total loss on toy feature maps (verified with NumPy)
import numpy as np
 
np.random.seed(1)
channels, spatial = 3, 4   # toy: 3 "feature channels", 4 "spatial locations"
 
# --- content loss: how far is the generated image from the content image? ---
content_feat = np.random.uniform(0, 1, size=(channels, spatial))
combo_feat = content_feat + np.random.normal(0, 0.1, size=(channels, spatial))
 
def content_loss(base, combination):
    return np.sum((combination - base) ** 2)
 
cl = content_loss(content_feat, combo_feat)
print("content loss:", round(cl, 4))
 
# --- style loss: how far is the generated image's Gram matrix from the style's? ---
def gram_matrix(x):
    return x @ x.T   # channels x channels correlation table
 
style_feat = np.random.uniform(0, 1, size=(channels, spatial))
combo_feat2 = style_feat + np.random.normal(0, 0.2, size=(channels, spatial))
 
def style_loss(style, combination, size):
    S, C = gram_matrix(style), gram_matrix(combination)
    return np.sum((S - C) ** 2) / (4.0 * (style.shape[0] ** 2) * (size ** 2))
 
sl = style_loss(style_feat, combo_feat2, spatial)
print("style loss:", round(sl, 6))
 
# --- combine with weights, exactly like the book's compute_loss ---
content_weight, style_weight = 0.25, 1.0
total = content_weight * cl + style_weight * sl
print("total loss:", round(total, 4))
text
content loss: 0.1043
style loss: 0.001504
total loss: 0.0276
text
content loss: 0.1043
style loss: 0.001504
total loss: 0.0276

Notice how much smaller the style loss’s contribution would be without a style_weightstyle_weight correction — in the real algorithm you tune content_weightcontent_weight, style_weightstyle_weight, and total_variation_weighttotal_variation_weight relative to each other until the result looks balanced; too much content weight and the output barely changes style, too much style weight and the original content dissolves.

Optimizing the generated image

Unlike every other model in this phase, there’s no model.fit()model.fit() here — you set combination_imagecombination_image up as a tf.Variabletf.Variable, compute gradients of the combined loss with respect to it, and apply them directly with an optimizer, for thousands of iterations:

The optimization loop (conceptual, following the book)
import tensorflow as tf
 
optimizer = tf.keras.optimizers.SGD(
    tf.keras.optimizers.schedules.ExponentialDecay(
        initial_learning_rate=100.0, decay_steps=100, decay_rate=0.96
    )
)
 
# combination_image = tf.Variable(preprocess_image(base_image_path))
# for i in range(1, 4001):
#     with tf.GradientTape() as tape:
#         loss = compute_loss(combination_image, base_image, style_reference_image)
#     grads = tape.gradient(loss, combination_image)
#     optimizer.apply_gradients([(grads, combination_image)])
The optimization loop (conceptual, following the book)
import tensorflow as tf
 
optimizer = tf.keras.optimizers.SGD(
    tf.keras.optimizers.schedules.ExponentialDecay(
        initial_learning_rate=100.0, decay_steps=100, decay_rate=0.96
    )
)
 
# combination_image = tf.Variable(preprocess_image(base_image_path))
# for i in range(1, 4001):
#     with tf.GradientTape() as tape:
#         loss = compute_loss(combination_image, base_image, style_reference_image)
#     grads = tape.gradient(loss, combination_image)
#     optimizer.apply_gradients([(grads, combination_image)])

A high initial learning rate (100) makes fast early progress, decaying by 4% every 100 steps so the optimization settles down as it approaches a good result — the same warm-up-then-cool-down idea you’ve seen with learning-rate schedules elsewhere.

Visualize it

Drag the slider (or watch it auto-animate) to blend a content pattern and a style pattern — at 0% it’s pure content layout, at 100% it’s pure style texture, and in between you get exactly what neural style transfer aims for: the content’s shapes rendered in the style’s texture:

sketch Blending content layout with style texture p5.js
Left grid: content shapes. Right grid: style texture. Center: a blend, sliding from pure content (0%) to pure style (100%) and back — the effect neural style transfer optimizes for.

Mini-checkpoint

Why does the style loss use the Gram matrix instead of comparing activations pixel-by-pixel like the content loss does?

  • Because style is about texture and correlation between features, not exact spatial layout — comparing raw activations pixel-by-pixel (like content loss does) would force the generated image to have the same arrangement as the style image, which is exactly what you don’t want. The Gram matrix summarizes which feature detectors fire together, discarding where they fire, so matching it transfers texture while leaving the content image’s layout alone.

🧪 Try It Yourself

Exercise 1 – Compute the Content Loss

Exercise 2 – Build the Gram Matrix

Exercise 3 – Combine Content and Style Loss with Weights

Next

That wraps up Phase 6. Continue to Introduction to Reinforcement Learning to start Phase 7 — instead of learning to generate data, an agent now learns a policy through trial, error, and reward.

If this helped you, consider buying me a coffee ☕

Buy me a coffee

Was this page helpful?

Let us know how we did