GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models

Before GLIDE, two families of models had made exciting but separate progress. On one side, diffusion models (like those in "Diffusion Models Beat GANs") had achieved photorealistic image quality on ImageNet — producing faces, animals, and scenes that humans could barely distinguish from real photographs. But they were class-conditional: you could ask for "dog" or "car," not "a corgi wearing a red bowtie and a purple party hat."

On the other side, DALL-E showed that an autoregressive model trained on text-image pairs could compose complex scenes from arbitrary text. But the images were often blurry and lacked the photorealism of diffusion models. The best DALL-E samples required generating hundreds of candidates and using CLIP to rerank them — an expensive post-processing step.

The question was straightforward: could you marry diffusion's image quality with DALL-E's text flexibility?

A diffusion model works in two phases. The forward process gradually destroys a real image by adding Gaussian noise over $T$ timesteps until it becomes pure static. The reverse process learns to undo this destruction: a neural network $\epsilon_\theta$ predicts the noise that was added at each step, and subtracts it to recover the image.

Think of it as learning to unscramble: if you know exactly what noise was stirred in, you can stir it back out. The model trains by seeing many noised images and learning to predict the noise component $\epsilon$ from the noisy version $x_t$.

GLIDE's core idea is simple: take a powerful diffusion model and give it text understanding. The architecture has three components working together:

1. A text encoder — a Transformer with 24 layers and width 2048, roughly 1.2 billion parameters. It converts a text caption into a sequence of $K$ token embeddings.

2. A visual diffusion model — based on the ADM (Ablated Diffusion Model) U-Net architecture, scaled to 512 channels and roughly 2.3 billion parameters. This is the workhorse that denoises images.

3. Two injection points — the text enters the visual model in two ways. First, the final token embedding replaces the class label (where a class-conditional model would use "dog" or "car"). Second, the full sequence of $K$ token embeddings is projected and concatenated to the attention context at every attention layer. This gives each layer of the U-Net access to the full text description via cross-attention.

Together, the model has 3.5 billion parameters — the base model — plus a separate 1.5 billion parameter upsampling diffusion model that takes the 64×64 output and increases it to 256×256 resolution.

The most impactful contribution of GLIDE is demonstrating that classifier-free guidance dramatically outperforms CLIP guidance for text-to-image diffusion. But what is classifier-free guidance, and why does it work so well?

The intuition begins with a mental model. During sampling, the model produces two predictions at each denoising step: one conditioned on the text caption $c$, and one unconditional (as if no caption were provided, using an empty prompt $\varnothing$). The difference between these two predictions tells the model: "this is the direction that makes the image more like what the text describes." By amplifying that difference, you push the image harder toward the text.

Think of it as navigation with two compasses. One compass points toward "any image" (the unconditional prediction). The other points toward "an image matching this description" (the conditional prediction). The vector from the first to the second is the "text direction." Multiplying that vector by a guidance scale $s > 1$ makes the model follow the text more aggressively — at the cost of some diversity.

The alternative guidance approach uses CLIP as an external evaluator. CLIP has two encoders — one for images $f(x)$ and one for text $g(c)$ — and outputs a similarity score. Higher dot product $f(x) \cdot g(c)$ means the image better matches the caption.

To use CLIP for guidance, at each denoising step the model computes the gradient of the CLIP similarity with respect to the current noisy image $x_t$. This gradient tells the model: "adjusting the image in this direction will increase its similarity to the text." The denoising mean is then shifted by this gradient, scaled by a guidance coefficient.

One critical detail: standard CLIP models were trained on clean images, not noisy ones. The noisy intermediate images during diffusion sampling are out-of-distribution for standard CLIP. GLIDE addresses this by training a noised CLIP model that explicitly processes noisy images $f(x_t, t)$, making the gradients more accurate.

GLIDE ran a systematic comparison between the two guidance strategies. On automated metrics, CLIP guidance achieves higher CLIP scores — but this is misleading. The authors hypothesized that CLIP guidance finds adversarial examples for the CLIP evaluator: images that score high on CLIP similarity without actually looking better to humans.

The definitive answer came from human evaluation. Evaluators were shown pairs of images and asked to choose which was more photorealistic or better matched the caption. The results were decisive: classifier-free guidance won in both photorealism and caption similarity. When comparing GLIDE with classifier-free guidance against DALL-E (even with DALL-E using expensive CLIP reranking of 512 candidates), human evaluators preferred GLIDE 87% of the time for photorealism and 69% for caption similarity.

On MS-COCO, GLIDE achieved a zero-shot FID of 12.24 — competitive with models explicitly trained on COCO — using a model roughly 3× smaller than DALL-E (3.5B vs. 12B parameters).

GLIDE's second major capability is image inpainting — filling in erased regions of an existing image guided by text. The user masks a region (e.g., a blank area above a couch), provides a text prompt ("a painting of a corgi on the wall"), and the model fills in that region with content matching the prompt while respecting the surrounding context.

To achieve this, the model is fine-tuned with four additional input channels: the masked RGB image (3 channels) plus a binary mask channel. Random regions of training images are erased, and the model learns to reconstruct them conditioned on the surrounding pixels and the text caption. The additional input weights are initialized to zero so the model starts from its pre-trained capabilities.

The results are remarkable: the model produces shadows, reflections, and lighting that match the surrounding scene. It can even match artistic styles when editing paintings. Users can iteratively build complex scenes — starting with a generated room, then adding furniture, decorations, and details one text prompt at a time.

The authors recognized that GLIDE could generate convincing fake images and enable deepfakes. Their response was multi-layered: they trained a smaller 300M parameter model (GLIDE filtered) on a dataset filtered to remove people, violence, and hate symbols. They released this smaller model publicly while retaining the full model.

Notably, the filtering process revealed biases: the model produced stereotypically pink toys for "toys for girls" versus gender-neutral results for "toys for boys," and tended to generate church-like buildings for "a religious place." These biases were amplified by classifier-free guidance. The authors were transparent about these limitations, setting an important precedent for responsible AI disclosure.