Consistency Models

Score-based diffusion models generate data by reversing a noise-injection process. Starting from pure Gaussian noise, they iteratively remove noise via a learned score function — the gradient of the log-probability density. This reverse process follows a Stochastic Differential Equation (SDE), or its deterministic counterpart, the Probability Flow ODE (PF-ODE).

The PF-ODE is especially important: it defines a deterministic trajectory from any noisy sample back to a clean data point. If you could solve this ODE exactly, you would get a perfect sample. But solving it requires many small steps, each one calling the neural network. A typical diffusion model needs 50–1000 forward passes to generate a single image.

Faster ODE solvers like DDIM and DPM-Solver reduce this to 10–50 steps, but going below 10 steps degrades quality sharply. Progressive distillation trains a student model to match two teacher steps in one, halving the step count each round. But it requires multiple rounds of training and inherits errors from each stage.

What if a model could learn to jump from any point on the ODE trajectory directly to the endpoint — in a single step?

The PF-ODE defines a unique trajectory for every data point. Given a clean image x₀, adding noise at level t produces x_t. The ODE trajectory connects every noisy version x_t back to the same x₀. This means that for any two noise levels t and t' on the same trajectory, the destination is identical.

A consistency function f captures this: f(x_t, t) = f(x_t', t') for all t and t' along the same trajectory. In other words, the function's output is consistent — it does not depend on where along the trajectory you evaluate it. This is the self-consistency property.

If we can train a neural network to satisfy this property, then at generation time we can sample noise x_T, evaluate the network once to get f(x_T, T), and obtain a clean sample directly. One step. No iteration.

A valid consistency function must satisfy one hard constraint: at the minimum noise level ε (close to zero), the function must return the input itself — f(x, ε) = x. This is the boundary condition. It ensures the function is anchored at the clean-data end of the trajectory.

To enforce this by construction, the model is parameterized as a weighted combination of the input and a neural network output. Two differentiable scalar functions control the mix: c_skip(t) and c_out(t). At t = ε, the skip connection dominates (c_skip(ε) = 1, c_out(ε) = 0), so the output equals the input. As t grows, the network's output takes over.

The first training approach leverages a pre-trained diffusion model as a teacher. The idea is straightforward: take a noisy sample at time step t₍ₙ₊₁₎, use the teacher's ODE solver to estimate where it would land one step earlier at tₙ, then train the consistency model so its output at both points matches.

Concretely, the training loss enforces that the student's consistency output at the noisier point approximately equals the EMA target network's output at the cleaner point. The teacher provides the ODE step that connects them. The EMA target network θ⁻ — an exponential moving average of the student's weights — stabilizes training, similar to how target networks work in deep reinforcement learning.

As the time discretization gets finer (N → ∞), the distillation loss converges to zero for a perfect consistency function. In practice, N is increased gradually during training — a curriculum that starts with coarse time steps and progressively refines.

The most surprising result of this paper is that consistency models can be trained entirely from scratch, without any pre-trained diffusion model. The trick is replacing the teacher's ODE step with an unbiased estimate of the score function.

Recall that the score function ∇ log p_t(x_t) can be estimated from a single training sample: it equals −(x_t − x₀) / t². This means we can construct adjacent points on the ODE trajectory using only the ground-truth data and random noise — no teacher required.

The consistency training loss has the same structure as the distillation loss, but instead of relying on the teacher's ODE solver, it uses the analytical relationship between x₀, noise ε, and the forward diffusion process. As the discretization refines (N → ∞), the training objective provably converges to a form whose minimizer is the true consistency function.

This is significant: consistency models are not just a distillation trick — they are a standalone family of generative models with their own training objective.

While one-step generation is the headline feature, consistency models can also do multistep sampling to improve quality. The idea is simple: generate a rough sample in one step, add noise back to an intermediate level, then denoise again. Each denoise-then-renoise cycle refines the output. With just 2–3 steps the quality improves significantly, offering a smooth compute-quality trade-off.

This same mechanism enables zero-shot image editing without any task-specific training. For inpainting, mask out a region, add noise, and denoise — the consistency model fills in the masked area coherently. For colorization, start with a grayscale image as partial information and let the model hallucinate colors. For super-resolution, begin with a low-resolution image, add noise, and denoise at higher resolution. All of these use the same trained model with no fine-tuning.

Consistency Distillation achieved an FID of 3.55 on CIFAR-10 and 6.20 on ImageNet 64×64 for one-step generation — a new state of the art among distillation methods. With two steps, CIFAR-10 FID dropped further to 2.93, approaching the quality of the teacher model running 100+ steps.

Consistency Training (without any teacher) achieved FID 7.5 on CIFAR-10 for one-step generation, outperforming all existing one-step non-adversarial generators. On LSUN bedrooms and cats at 256×256, consistency models produced compelling samples with 1–3 steps.

Importantly, the same model handles generation and editing. Zero-shot inpainting, colorization, and super-resolution all worked with no task-specific training, using only the multistep sampling procedure.

Consistency models use the same U-Net backbone as the teacher diffusion model — no architectural changes are needed. The time-conditioning input simply indicates the noise level. The key training decisions are:

Distance metric d(·,·): The paper explored L2, L1, and LPIPS (Learned Perceptual Image Patch Similarity). LPIPS consistently performed best, as it measures perceptual similarity rather than pixel-by-pixel error — important because the mapping from noise to data is one-to-many.

EMA decay rate μ: The target network θ⁻ is an exponential moving average of θ. A schedule that slowly increases μ toward 1 during training worked best.

Schedule function N(k): The number of discretization steps N is increased during training according to a schedule. Starting coarse and refining avoids early instability while ensuring fine-grained consistency at convergence.

ODE solver choice (for CD): Euler and Heun solvers both work. Heun (a second-order solver) gives slightly better results for the same discretization.

Consistency models catalyzed a paradigm shift: from "diffusion models are slow but high-quality" to "diffusion-quality generation can be fast." The self-consistency principle proved to be more than a clever trick — it is a fundamental design paradigm for generative models that trade iterative refinement for direct prediction.