Analyzing and Improving the Image Quality of StyleGAN

StyleGAN uses Adaptive Instance Normalization (AdaIN) to inject style information into each layer of the generator. AdaIN works by normalizing each feature map to zero mean and unit variance, then scaling and shifting it using parameters derived from the style vector. The concept is elegant: the content of the feature map is preserved, but its statistical signature — its "style" — is overwritten by the latent code.

But the authors discovered a hidden side effect. When the generator wants to encode a strong signal in one feature map, instance normalization washes it away because it forces unit variance. The generator learns a workaround: it creates a single dominant spike — a localized pixel with enormous value — that takes over the statistics of the entire feature map. After normalization, this spike is suppressed in magnitude, but it has already distorted the mean and variance, effectively letting the generator sneak its desired signal through the normalization gate. These spikes manifest as the characteristic "water droplet" artifacts visible in generated images.

Think of it like a classroom where the teacher forces everyone to speak at the same volume. One student wants to shout — so they create a distraction (the spike) that throws off the teacher's volume measurement, letting them get their message through. The distraction is the artifact.

The key insight is: why normalize the activations after convolution when you can normalize the convolution weights before it? If the weights themselves are scaled so that the output has unit variance, there is no need for instance normalization — and no opportunity for the generator to create spike artifacts.

The solution has two steps. Modulation scales the convolution weights by the style vector, exactly like AdaIN would scale the activations — but it does so on the weights, not on the feature map. Demodulation then normalizes the modulated weights so that the output feature map will have approximately unit standard deviation, assuming the input has unit variance.

The mental model: imagine a pipeline where water (the signal) flows through adjustable valves (convolution weights). AdaIN adjusts the flow after the water passes through — it measures the water pressure and forces it to a target level. But by then, the pressure spikes have already damaged the pipes. Weight demodulation adjusts the valves before the water flows — the pressure is correct from the start, so no damage occurs.

Formally, let $w_{i,j,k}$ be the original convolution weights where $i$ indexes input channels, $j$ indexes output channels, and $k$ indexes spatial positions (kernel elements). The style vector provides a per-layer scaling factor $s_i$ for each input channel.

Step 1 — Modulation: Scale each weight by the corresponding style factor:

Step 2 — Demodulation: Normalize the modulated weights so the output has unit standard deviation:

Progressive growing trains the generator by starting at low resolution (4×4) and gradually adding higher-resolution layers during training. This was a breakthrough in ProGAN and StyleGAN for stabilizing high-resolution training. But the authors of StyleGAN2 noticed a subtle consequence: facial features like eyes and teeth have a strong preference for specific pixel locations on the canvas.

When interpolating between two latent codes — smoothly transitioning from one face to another — features that should move (because the face is turning) instead stay anchored to fixed positions. The eyes slide into place at the same coordinates regardless of the face's pose. The effect is like a puppet show where the eyes are painted on the backdrop rather than on the puppet: the puppet moves, but the eyes do not.

The cause is that progressive growing trains each resolution independently at first. The lower-resolution layers learn to place features at specific absolute positions, and these become "anchor points" that the higher-resolution layers cannot override. The features are stuck because the early layers set where they go, and the later layers just refine details without moving the positions.

StyleGAN2 replaces progressive growing with a fixed-depth architecture that trains all resolutions from the start. The generator uses skip connections — each resolution contributes to the final image via bilinear upsampling and addition. The discriminator uses residual connections with downsampling. This design is inspired by the MSG-GAN approach.

The crucial difference: with progressive growing, the 4×4 layer has a monopoly on structure during early training. With skip connections, every layer contributes from the beginning, and the balance of contributions shifts organically during training rather than being imposed by a rigid schedule.

Even after fixing the artifacts, there remains a deeper question: how well-behaved is the mapping from latent codes to images? Ideally, if you take a small step in any direction in the latent space $\mathcal{W}$, the corresponding change in the generated image should be proportional and consistent — not wildly different depending on where you are or which direction you step.

A generator with this property is called well-conditioned. Well-conditioning matters for two reasons. First, it makes latent space interpolation smooth and meaningful — walking between two faces produces a natural morph rather than jarring jumps. Second, it makes the generator invertible: given a real image, you can find the latent code that produced it, because the mapping does not fold or collapse.

The authors measure conditioning by looking at the Jacobian $\mathbf{J}_w$ of the generator — the matrix of partial derivatives that describes how each pixel changes in response to each latent dimension. Good conditioning means this Jacobian has consistent scale everywhere: a step of length $\delta$ in $\mathcal{W}$ always produces an image change of roughly the same magnitude.

Computing the path length regularizer (and the R1 gradient penalty for the discriminator) at every training step is expensive — it requires a full backward pass through the generator just for the regularization term. The authors discovered that these regularizers change slowly during training, so computing them every step is wasteful.

Lazy regularization computes the regularization term once every $k$ mini-batches (they use $k = 16$) and scales the regularization weight by $k$ to compensate. This reduces training cost significantly with virtually no impact on final quality. The intuition: regularizers guide the landscape, and the landscape shifts slowly. You do not need to re-measure the terrain every step — every 16 steps is frequent enough to keep the training on track.

The authors introduced a diagnostic to measure whether the generator actually uses the full output resolution or concentrates its detail at lower frequencies. By masking different frequency bands and measuring the perceptual impact, they found that the original StyleGAN underutilizes its higher resolutions — much of the fine detail is actually determined at lower layers.

This motivated training larger models (Config F → Config E → Config F with more channels) and confirmed that StyleGAN2's architectural changes let the generator distribute detail more effectively across all resolution levels. The final model achieves an FID of 2.84 on FFHQ at 1024×1024 resolution, a significant improvement over StyleGAN's 4.40.

Because path length regularization makes the generator well-conditioned, it becomes possible to project a given image back into the latent space — finding the latent code $\mathbf{w}$ that, when passed through the generator, reproduces the image as closely as possible. This is done by optimizing over $\mathbf{w}$ to minimize the perceptual loss (LPIPS) between the generated and target images.

This projection has a practical application: network attribution. If you suspect an image was generated by a specific GAN, project it into that GAN's latent space. If the projection is successful (low reconstruction error), the image likely came from that network. If projection fails (high error), the image either is real or came from a different generator.

The well-conditioned generator produced by StyleGAN2 makes this projection far more reliable than with StyleGAN, because the smooth latent space means the optimizer can find the correct code without getting stuck in local minima.

The complete StyleGAN2 pipeline retains the two-stage structure of StyleGAN. A mapping network $f$ transforms the random input $\mathbf{z} \sim \mathcal{N}(0, I)$ into an intermediate latent code $\mathbf{w} \in \mathcal{W}$. The synthesis network $g$ then converts $\mathbf{w}$ into an image, with each layer receiving its own style derived from $\mathbf{w}$ via learned affine transformations.

What changes in StyleGAN2 relative to StyleGAN:

• AdaIN is replaced by weight modulation + demodulation — style is injected through the weights, not through activation statistics.
• Progressive growing is replaced by a skip-connection generator and a residual discriminator — all resolutions train simultaneously.
• Path length regularization is added to the generator loss — the Jacobian is encouraged to have uniform scale.
• Lazy regularization computes regularizers every 16 steps — saving compute with no quality loss.
• Noise inputs and biases are moved outside the style block — a minor reorganization that simplifies the architecture.

StyleGAN2's legacy is paradoxical: it set the quality bar so high that it motivated the very class of models — diffusion — that eventually displaced GANs from the top of the leaderboard. Yet StyleGAN2's architectural lessons endure. Weight demodulation influenced normalization design in subsequent architectures. Path length regularization established the importance of well-conditioned generators. And the idea that image quality can be improved by diagnosing and surgically removing specific artifacts — rather than simply scaling up — remains a powerful principle in generative modeling.