Diffusion Models Beat GANs on Image Synthesis

A diffusion model generates images by reversing a gradual noising process. During training, real images are progressively corrupted by adding Gaussian noise across $T$ timesteps, until the image becomes pure static. The model learns to undo this corruption one step at a time — predicting and removing the noise component at each timestep.

Think of it like a highway with $T$ exits. At exit 0, you have a clean image. At exit $T$, you have pure noise. The forward process drives the image from exit 0 to exit $T$, adding a little noise at each exit. The model learns the reverse trip: starting from pure noise at exit $T$ and gradually denoising back to exit 0.

The noise predictor $\epsilon_\theta(x_t, t)$ takes a noisy image $x_t$ and the current timestep $t$ as input, and predicts the noise $\epsilon$ that was added. Training uses a simple mean squared error loss between the predicted noise and the actual noise.

The first hypothesis: the gap between diffusion models and GANs comes partly from under-optimized architectures. GAN architectures like StyleGAN2 and BigGAN had been refined through years of ablation studies. Diffusion models were still using the original U-Net from DDPM with minimal changes.

Dhariwal and Nichol conducted a systematic ablation study on ImageNet 128×128, testing one change at a time. They discovered that several modifications compound to give a substantial boost in FID, and most of these are inspired by best practices from GAN and Transformer literature.

The key modifications, each tested independently and then combined, are:

More depth vs. width. Increasing depth (more residual blocks per resolution) while holding model size constant improves FID, but slows training. Wider models reach the same quality faster, so the authors chose width over depth.

Multi-resolution attention. Instead of applying attention only at 16×16 resolution (as in DDPM), the improved model applies attention at 32×32, 16×16, and 8×8 resolutions. This helps the model capture both fine details and global structure.

More attention heads. Increasing from 1 head to multiple heads with 64 channels per head significantly improves FID. This matches modern Transformer practice, where many small heads outperform few large ones.

BigGAN residual blocks. Using BigGAN-style upsampling and downsampling residual blocks instead of simple convolutions improves information flow through the network.

Adaptive Group Normalization (AdaGN). Instead of adding timestep and class embeddings via simple addition, AdaGN modulates the normalization parameters: $\text{AdaGN}(h, y) = y_s \cdot \text{GroupNorm}(h) + y_b$, where $y_s$ and $y_b$ come from a linear projection of the timestep and class embedding. This is analogous to adaptive instance normalization from StyleGAN.

The second insight addresses a fundamental advantage that GANs had: they could trade off diversity for fidelity using the truncation trick. A smaller truncation range produces sharper but less diverse images. Diffusion models had no equivalent knob.

The solution: classifier guidance. The idea is beautifully simple. Train a separate classifier $p_\phi(y|x_t, t)$ on noisy images at every noise level. During sampling, use the classifier's gradients to nudge each denoising step toward a target class $y$. The classifier acts like a museum guide whispering directions to the restorer: "this should look more like a corgi."

Mathematically, the conditional reverse process becomes:

Under reasonable assumptions (the classifier's log-probability has low curvature compared to the precision of the Gaussian transition), this product simplifies elegantly. The conditional transition remains a Gaussian — like the unconditional one — but with its mean shifted by $\Sigma g$, where $g = \nabla_{x_t} \log p_\phi(y|x_t)$ is the classifier gradient and $\Sigma$ is the predicted covariance.

The sampling algorithm is simple: at each timestep, compute the unconditional mean $\mu$ and variance $\Sigma$, then sample from $\mathcal{N}(\mu + s\Sigma g, \Sigma)$, where $s$ is a gradient scale factor.

A key practical discovery: using the classifier gradients at their natural scale ($s = 1$) is not enough. While the classifier assigns around 50% probability to the target class, the resulting images do not visually match the class. The gradients need to be amplified.

Scaling the gradients by $s > 1$ has a clean theoretical interpretation. It is equivalent to sampling from a sharpened classifier distribution $p(y|x)^s$. When $s > 1$, this distribution becomes more peaked — it concentrates probability on the most confident class predictions. Think of it as turning up the "conviction" of the museum guide.

The authors found that the scale can be increased by an order of magnitude (up to $s = 10$) without producing adversarial examples. This smoothly trades off recall (diversity) for precision (fidelity), with FID reaching its optimum at an intermediate scale.

The improved architecture alone (ADM) achieved state-of-the-art FID on LSUN bedrooms (1.90), horses (2.57), and cats (5.57) — all unconditional, beating StyleGAN2. On ImageNet 64×64 the improved model reached FID 2.07.

With classifier guidance (ADM-G), diffusion models surpassed BigGAN-deep on ImageNet at every resolution: FID 2.97 at 128×128, 4.59 at 256×256, and 7.72 at 512×512. Crucially, the diffusion models maintained higher recall (distribution coverage) than GANs at comparable or better precision.

Even with only 25 sampling steps using DDIM, the guided model matched BigGAN-deep's FID while still covering more of the distribution.

Finally, combining classifier guidance with upsampling diffusion models gave the best overall results: FID 3.94 on 256×256 and 3.85 on 512×512. The authors showed that guidance and upsampling improve quality along different axes — guidance boosts fidelity while upsampling improves spatial resolution — so they complement each other.

The Gaussian perturbation derivation above only works for stochastic sampling. For deterministic samplers like DDIM, the authors use a score-based trick. The noise predictor $\epsilon_\theta(x_t)$ can be reinterpreted as a scaled score function:

$\nabla_{x_t} \log p_\theta(x_t) = -\frac{1}{\sqrt{1-\bar\alpha_t}} \epsilon_\theta(x_t)$

Adding the classifier gradient gives the joint score:

$\nabla_{x_t} \log(p_\theta(x_t) \cdot p_\phi(y|x_t)) = \nabla_{x_t}\log p_\theta(x_t) + \nabla_{x_t}\log p_\phi(y|x_t)$

This translates into a modified noise prediction $\hat\epsilon(x_t) = \epsilon_\theta(x_t) - \sqrt{1-\bar\alpha_t}\,\nabla_{x_t}\log p_\phi(y|x_t)$ that can be plugged directly into the DDIM sampler. The classifier gradient literally "subtracts class-relevant noise" from the prediction, so the denoiser favors the target class.

The authors also explored two-stage upsampling stacks, where a low-resolution diffusion model generates a small image and then a separate upsampling model enlarges it. They found that guidance and upsampling improve quality along different axes:

Upsampling improves spatial resolution and precision while maintaining high recall. Guidance improves class consistency and fidelity but reduces recall. When combined — applying guidance only to the low-resolution model before upsampling — the two approaches complement each other, yielding the best overall FIDs: 3.94 on 256×256 and 3.85 on 512×512.

This is like having a specialist who draws small but perfectly class-accurate sketches (guidance) and then a different specialist who enlarges sketches into detailed paintings (upsampling). Each handles a different aspect of quality.

This paper was a watershed moment for generative modeling. By proving that diffusion models could match and exceed GANs on their own turf — raw image quality — it shifted the entire field's center of gravity. Within months, the diffusion paradigm became dominant.

The most direct descendant is classifier-free guidance (Ho & Salimans, 2021). It observed that training a classifier on noisy images is cumbersome and limits the approach to labeled data. Their elegant alternative: train a single model that can operate both conditionally and unconditionally, then interpolate between the two during sampling. This preserved the diversity-fidelity trade-off without needing a separate classifier, and became the default guidance method in virtually all subsequent text-to-image models.

GLIDE (Nichol et al., 2021) applied both classifier guidance and classifier-free guidance to text-conditional image generation, showing that guided diffusion models could produce images from text prompts — laying the groundwork for DALL·E 2, Imagen, and Stable Diffusion.