Denoising Diffusion Probabilistic Models

Before DDPM, two families of generative models dominated image synthesis:

GANs (Generative Adversarial Networks) pitted a generator against a discriminator in a minimax game. They produced sharp images but suffered from training instability: mode collapse (the generator learns to produce only a few outputs), vanishing gradients for the generator, and extreme sensitivity to hyperparameters. Scaling GANs required years of engineering tricks.

VAEs (Variational Autoencoders) encoded images into a latent space and decoded them back. Training was stable and principled (maximizing a variational lower bound), but the decoded images were systematically blurry — the model averaged over possible outputs rather than committing to one sharp reconstruction.

Diffusion models had been proposed theoretically by Sohl-Dickstein et al. (2015), but nobody had shown they could produce competitive image quality. The DDPM paper changed that.

The forward process is a predetermined recipe — no learning happens here. Starting from a clean image $x_0$, we add a tiny amount of Gaussian noise at each step, producing a sequence $x_1, x_2, \ldots, x_T$. After enough steps ($T \approx 1000$), the image is indistinguishable from pure static.

Think of it as slowly turning up the static on a television: at step 1 you can still see the picture clearly; by step 500 it is getting fuzzy; by step 1000 it is pure snow.

Each step is a Markov transition — the amount of noise added at step $t$ depends only on the image at step $t-1$, not on any earlier step. This Markov property is what makes the math tractable.

The sequence $\beta_1, \beta_2, \ldots, \beta_T$ is the noise schedule — it controls how quickly the image is destroyed. Ho et al. used a linear schedule from $\beta_1 = 10^{-4}$ to $\beta_T = 0.02$ with $T = 1000$ steps.

Why does the schedule matter? If noise is added too fast, the reverse process must make large corrections and is prone to error. If too slow, training is inefficient and sampling takes forever. The linear schedule strikes a balance: gentle at the start (preserve structure), accelerating at the end (finish destroying the signal).

Later work (Improved DDPM, Nichol & Dhariwal 2021) explored cosine schedules that keep the signal-to-noise ratio dropping more uniformly, but the original linear schedule was sufficient for state-of-the-art results.

The reverse process is where the magic happens — and where all the learning occurs. Starting from pure noise $x_T \sim \mathcal{N}(0, \mathbf{I})$, the model iteratively denoises: $x_T \to x_{T-1} \to \ldots \to x_1 \to x_0$.

At each step, the model must answer: "Given this noisy image $x_t$ and the current timestep $t$, what does a slightly less noisy version $x_{t-1}$ look like?" This is parameterized as a Gaussian:

Instead of predicting $\mu_\theta$ directly, Ho et al. made a key design choice: reparameterize the mean in terms of the predicted noise $\epsilon_\theta(x_t, t)$. The network takes a noisy image and the timestep, and outputs its best guess of the noise that was added. The mean is then computed from this prediction:

The full variational lower bound (ELBO) for diffusion models involves a sum of KL divergences across all timesteps — complex and expensive. Ho et al.'s breakthrough was showing that a drastically simplified loss works just as well (and often better):

1. Sample a clean image $x_0$ from the dataset
2. Sample a random timestep $t \sim \text{Uniform}(1, T)$
3. Sample random noise $\epsilon \sim \mathcal{N}(0, \mathbf{I})$
4. Construct the noisy image: $x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon$
5. Ask the network to predict the noise: $\epsilon_\theta(x_t, t)$
6. Compute the loss: $\|\epsilon - \epsilon_\theta(x_t, t)\|^2$

That is it. Mean squared error between the true noise and the predicted noise. No adversarial games, no reconstruction-vs-KL balancing, no instability. Just: "can you guess what noise I added?"

The neural network $\epsilon_\theta$ that predicts the noise is a U-Net — an encoder-decoder architecture with skip connections, originally designed for biomedical image segmentation. The U-Net is ideal for denoising because it operates at the same resolution as the input (unlike a classifier which compresses to a single label) and its skip connections preserve fine spatial details across the encoding and decoding stages.

DDPM augments the basic U-Net with two critical additions:

Sinusoidal timestep embeddings — the network must know how noisy the input is. The timestep $t$ is encoded using sinusoidal positional embeddings (the same idea from the Transformer) and injected into every residual block. This lets the same network handle all noise levels, from barely-touched ($t = 1$) to pure static ($t = T$).

Self-attention at lower resolutions — at the 16×16 resolution level, self-attention layers let distant spatial regions communicate, improving global coherence (e.g., both eyes facing the same direction in a face).

DDPM's simplified loss hides a deep connection to score-based generative models. The "score" of a distribution is the gradient of its log-probability: $\nabla_x \log p(x)$. It points toward regions of higher density — "which direction should I move to make this sample more likely?"

The noise-prediction network $\epsilon_\theta$ is implicitly learning a scaled version of the score function. Specifically: $\epsilon_\theta(x_t, t) \approx -\sqrt{1 - \bar\alpha_t}\, \nabla_{x_t} \log q(x_t)$.

This means DDPM's reverse process is closely related to Langevin dynamics — a physics-inspired sampling method where you follow the score (the gradient) with added noise to draw samples from a distribution. Song & Ermon (2019) had shown this approach works; Ho et al. showed the diffusion framework achieves it naturally and produces better samples.

This connection is what makes diffusion models so theoretically grounded: they are simultaneously variational inference models (optimizing a bound on likelihood), denoising autoencoders (predicting clean from noisy), and score-matching models (learning the gradient of the data distribution).

Once trained, generating an image follows the reverse process:

1. Sample $x_T \sim \mathcal{N}(0, \mathbf{I})$ — start from pure noise
2. For $t = T, T-1, \ldots, 1$: — Predict $\epsilon_\theta(x_t, t)$ — Compute $\mu_\theta(x_t, t)$ using the formula above — Sample $x_{t-1} \sim \mathcal{N}(\mu_\theta, \sigma_t^2 \mathbf{I})$
3. Return $x_0$ — the generated image

This is the ancestral sampling algorithm. It requires running the full T = 1000 steps, making it slow compared to a single-pass GAN. However, each step is a simple forward pass through the U-Net, and the quality of the result is remarkable.

On unconditional image generation on CIFAR-10, DDPM achieved an FID score of 3.17 — matching ProgressiveGAN which held the state of the art at the time. On LSUN (bedrooms, churches, horses), it produced diverse, high-fidelity 256×256 samples.

Crucially, DDPM achieved this without any adversarial training — no discriminator, no minimax game, no mode collapse. Training was as stable as training a classifier: just gradient descent on the simple loss.

The paper also demonstrated that diffusion models naturally produce a progressive lossy decompression scheme: each reverse step reveals more detail, from coarse structure to fine texture, analogous to how progressive JPEG decoding works.

From a single idea — "learn to predict and remove noise" — an entire ecosystem emerged. Every major text-to-image, video generation, and 3D synthesis system today traces its lineage back to this paper. DDPM proved that patience (1000 small steps) beats brute force (one-shot adversarial generation), and that simplicity in training objectives can yield extraordinary results.