Generative Modeling by Estimating Gradients of the Data Distribution

Traditional generative models try to learn the probability density $p(x)$ — essentially, how tall the landscape is at each point. But computing $p(x)$ requires a normalization constant (the partition function) that sums over all possible data points — an intractable integral in high dimensions.

This paper takes a different approach: instead of learning the height, learn the slope. The score function is defined as the gradient of the log-density:

$s(x) = \nabla_x \log p(x)$

This is a vector field: at every point in data space, it points toward higher-density regions. Crucially, the gradient of the log-density does not depend on the normalization constant — when you take the gradient, the constant disappears because the derivative of a constant is zero. This means we can learn the score without ever computing the intractable partition function.

There's an apparent paradox: to train the network to approximate the true score $\nabla_x \log p(x)$, we would need to know $p(x)$ — which is exactly what we're trying to avoid. Score matching resolves this elegantly.

The naive objective would be to minimize the Fisher divergence — the expected squared distance between the model's output and the true score. Through integration by parts (Hyvärinen, 2005), this objective can be rewritten into a form that depends only on the model's Jacobian and values at data points — not on $p(x)$ itself.

The paper uses denoising score matching (Vincent, 2011), which is even simpler: perturb each data point $x$ with known Gaussian noise to get $\tilde{x}$. For a Gaussian perturbation, the true score of the noisy distribution has a closed form — it is simply the direction back to the clean data point, scaled by the noise variance:

The training loss is therefore refreshingly simple: add noise to a data point, ask the network to predict the score, and penalize the difference:

Real data occupies a thin manifold in high-dimensional space — images of faces lie on a curved surface far lower-dimensional than the pixel space. In vast regions between data clusters, the score is essentially undefined because there is virtually zero probability density there.

A score network trained with a single small noise level will be accurate near the data but unreliable in the empty regions. Langevin dynamics initialized from random noise would have to traverse these empty deserts with no reliable gradient signal — the random walker would wander aimlessly, never finding the data.

Think of it spatially: with a tiny noise level, each data point creates a narrow "cone of attraction." If your random hiker starts far away, none of the cones reach them — they feel no pull toward any valley.

The solution is elegant: use multiple noise levels. Large noise inflates each data point into a wide hill that covers the empty regions, giving the hiker a coarse direction. Small noise sharpens the landscape for precise positioning. The paper uses a geometric sequence of noise levels from large $\sigma_1$ to small $\sigma_L$.

Instead of training a separate score network per noise level, the paper trains one shared network conditioned on the noise level $\sigma$. The Noise Conditional Score Network (NCSN) $s_\theta(x, \sigma)$ takes both a (noisy) data point and the noise scale as input.

The combined training objective sums the denoising score matching loss across all noise levels, weighted by $\lambda(\sigma_i) = \sigma_i^2$ to balance the different scales:

Langevin dynamics is a sampling algorithm from physics: to draw samples from a distribution, start from random noise and iteratively take small steps in the direction of the score, plus a bit of random noise for exploration:

$x_{t+1} = x_t + \frac{\alpha}{2} s_\theta(x_t) + \sqrt{\alpha}\, z_t, \quad z_t \sim \mathcal{N}(0, I)$

Think of it as a ball rolling downhill on the log-density surface, but with random jitter so it doesn't get stuck in local bumps. Given enough steps and a small enough step size $\alpha$, the ball's position converges to a sample from the true distribution.

But plain Langevin dynamics with one noise level fails for the manifold reasons discussed above. The paper's key sampling innovation is annealed Langevin dynamics: run Langevin dynamics in stages. Start with the score at the largest noise $\sigma_1$ (wide smooth landscape), run $T$ steps to find the general region, then switch to $\sigma_2$ (slightly sharper landscape), run $T$ more steps, and so on — gradually reducing the noise until you reach $\sigma_L$ and settle precisely on the data manifold.

The noise levels $\{\sigma_i\}_{i=1}^{L}$ form a geometric sequence: each level is a fixed ratio of the previous one. The paper offers two design principles for selecting the endpoints.

The largest noise $\sigma_1$ should be large enough that the noisy distribution $q_{\sigma_1}(x) \approx \mathcal{N}(0, \sigma_1^2 I)$ is nearly indistinguishable from the prior — pure Gaussian noise. This ensures that Langevin dynamics at the first scale can mix well starting from anywhere in space.

The smallest noise $\sigma_L$ should be small enough that the noisy distribution $q_{\sigma_L}$ is nearly indistinguishable from the true data — so that the final samples are clean. In practice, $\sigma_L$ is set to a small fraction of the typical distance between nearest data points.

With $L = 10$ levels and a geometric ratio between these extremes, the sequence creates a smooth bridge from "everything is one big blob" to "sharp, realistic images."

On CIFAR-10, the NCSN achieved an Inception score of 8.87 and an FID of 25.32 — the best among all non-adversarial generative models at the time. For comparison, the best likelihood-based model (Glow) had an FID of 46.90, and the best GAN (SNGAN) had an FID of 21.70.

On CelebA (128×128), the model generated faces with fine details like hair strands and facial features — quality competitive with GANs but without the mode collapse or training instability that plagued adversarial methods.

Perhaps more important than the numbers: the model generated diverse samples. Unlike GANs, which often converge to a subset of modes, the score-based approach covers the full data distribution because Langevin dynamics is theoretically guaranteed to converge to the correct distribution given enough steps.

What Song and Ermon demonstrated was that we don't need adversarial training, invertible architectures, or explicit density computation to build powerful generative models. We just need to learn which direction is "downhill" — toward the data — and follow it. This simple, physics-inspired insight turned out to be the key that unlocked the diffusion revolution.