Image Style Transfer Using Convolutional Neural Networks

When we look at a photograph of a building painted by Monet, we effortlessly separate two things: the content (a building by a river) and the style (soft pastel colors, visible brushwork). But for a computer, an image is just a grid of numbers. Traditional image-processing tools had no principled way to tease these apart.

Previous texture-transfer methods used hand-designed statistics — histograms of oriented gradients, wavelet coefficients — that captured low-level patterns well but couldn't understand "there is a building here." So they either destroyed the objects in the photo or failed to transfer the painting's character.

The key insight: convolutional neural networks trained for object recognition already build internal representations that separate what is in an image from how it looks. We just need to find where each lives inside the network and how to extract it.

The authors use VGG-19, a 19-layer convolutional network trained on ImageNet for object recognition. They do not retrain it or add any layers — they simply pass images through it and read off the internal activations.

Recall that a CNN builds a hierarchy of feature maps: early layers detect edges and colors, middle layers detect textures and patterns, and deep layers detect entire objects and their arrangements. This hierarchy is exactly the lever we need:

• Content lives in the deep layers — where the network knows "this is a building" but has forgotten the exact pixel colors.
• Style lives in the correlations across feature maps at every level — the texture fingerprint that says "thick swirling strokes in blue and yellow" without caring where the strokes are.

Pass a photograph through VGG-19 and record the feature maps at a chosen deep layer (the paper uses conv4_2). These feature maps form the content representation: a set of activation patterns that encode the spatial arrangement of objects — buildings, sky, river — but have shed exact pixel values and textures.

Think of it like a detailed pencil sketch: the shapes and layout are all there, but the colors and brushwork are gone. To measure how close a generated image's content is to the original photo, we compare their feature maps at this layer.

Read it as a difference report: for every filter in the chosen layer, how different is the generated image's response from the photo's response? If the generated image has a building in the same place as the photo, those filter responses will match closely, even if the colors and textures are completely different.

Style is harder to pin down than content. A Van Gogh painting has a distinctive texture — swirling strokes, thick impasto, vivid complementary colors — that is present everywhere, not tied to any particular object.

The key idea: style is captured by which feature maps fire together. If, across the entire image, a "blue" filter and a "curved edge" filter tend to activate at the same locations, that correlation IS the style signal. The mathematical tool for capturing all such pairwise correlations is the Gram matrix.

To build the Gram matrix for a given layer: take all $N$ feature maps (each of size $M$ pixels), flatten each into a row vector, and compute the $N \times N$ matrix of dot products. Entry $(i,j)$ measures how much feature $i$ and feature $j$ co-activate across the image. The Gram matrix is essentially a correlation fingerprint of how different visual patterns relate to each other — the "DNA" of the artist's style.

Think of each feature map as a specialist reporter covering the image: one reports "edges here," another reports "blue here," a third reports "curves here." The Gram matrix records which reporters tend to file their stories from the same locations. In a Van Gogh painting, the "curves" reporter and the "blue" reporter frequently agree — that pattern of agreement is what makes the painting look like a Van Gogh.

Crucially, the Gram matrix sums over all spatial positions, so it captures what patterns co-occur without caring where. This is exactly what we want: style is location-independent.

The style loss compares the Gram matrices of the generated image against the Gram matrices of the style painting at multiple VGG layers (typically conv1_1 through conv5_1). Using multiple layers is critical: low layers capture fine textures (brush hair marks), middle layers capture medium patterns (swirl shapes), and deep layers capture large-scale style (color palette arrangements).

The total loss is a weighted sum of content loss and style loss. The algorithm starts from a white noise image and iteratively adjusts its pixels using gradient descent (specifically L-BFGS, a quasi-Newton optimizer) to minimize this combined loss.

The weights α and β control the trade-off: high α/β preserves the photograph's structure with subtle style hints; low α/β lets the painting's texture dominate, turning the photo into an almost abstract artwork.

This is not training a network — the VGG weights are frozen. We are optimizing the image itself. Each pixel is a variable, and gradient descent sculpts the pixel values until the image simultaneously looks like the photo (in terms of feature maps) and feels like the painting (in terms of Gram matrices).

Here is the complete algorithm step by step:

1. Pass the content photo through VGG-19 and record feature maps at conv4_2 → this is the content target.
2. Pass the style painting through VGG-19 and compute Gram matrices at conv1_1, conv2_1, conv3_1, conv4_1, conv5_1 → these are the style targets.
3. Initialize the output image from white noise (or from the content photo for faster convergence).
4. In each iteration: pass the output image through VGG-19, compute content loss against the content target and style loss against the style targets, sum them with weights α and β, backpropagate to compute gradients with respect to the pixels of the output image, and update the pixels using L-BFGS.
5. Repeat until convergence (typically a few hundred iterations).

This is a deep question the paper answers empirically but later work has formalized. The Gram matrix can be understood as matching the distribution of features across the image. Later research showed that minimizing the Gram matrix difference is equivalent to minimizing a specific form of Maximum Mean Discrepancy — a statistical distance between distributions.

Intuitively: two images have the same style if the "recipe" of visual ingredients is the same — the same proportions of edges, colors, and textures co-occurring, regardless of where they appear. The Gram matrix is exactly that recipe card.

The paper also shows that reconstructing an image from Gram matrices alone (without any content constraint) produces textures that look like the original painting — wavy blue and yellow strokes from Starry Night, for instance — confirming that Gram matrices truly encode style.

Several decisions shape the final result:

• Which content layer? Deeper layers preserve abstract structure; shallower layers preserve more exact pixel details. The paper uses conv4_2 — deep enough to capture objects but abstract enough to allow style freedom.
• Which style layers? Using only one layer captures style at one scale. Using all five (conv1_1 through conv5_1) captures everything from fine grain to broad composition.
• Initialization. Starting from white noise produces artistic results but converges slowly. Starting from the content photo converges faster and preserves more structure.
• Max pooling vs. average pooling. The authors found that replacing max pooling with average pooling slightly improves gradient flow and produces smoother results.