DALL·E: Zero-Shot Text-to-Image Generation

The core insight behind DALL·E is surprisingly simple: if we can turn an image into a sequence of discrete tokens — much like words in a sentence — then we can train a language model to predict those tokens given a text prompt. The model never "sees" pixels directly. Instead, it learns the joint probability of text tokens followed by image tokens, and generates images by sampling that distribution autoregressively — one token at a time, left to right.

This unifies text understanding and image generation into a single architecture: the transformer. No GANs, no separate text encoder feeding into a convolutional generator, no object detectors. Just a single stream of tokens and the same next-token prediction objective that powers GPT-3.

But there is a catch: a 256×256 color image has 196,608 pixel values. Modeling each pixel as a token would require a sequence length no transformer could handle. The solution is compression — and that is where the discrete VAE comes in.

The first stage of DALL·E trains a discrete Variational Autoencoder — the dVAE — on images alone, without any text. Its job is to learn a visual codebook: a dictionary of 8,192 visual "words," each representing a small patch of texture, color, and shape.

The dVAE encoder takes a 256×256 RGB image and compresses it into a 32×32 grid. At each of the 1,024 grid positions, the encoder outputs a probability distribution over the 8,192 codebook entries, asking: "which visual word best describes this patch?" The decoder then takes those 1,024 chosen codebook vectors and reconstructs the full 256×256 image.

Think of it like a mosaic artist who has 8,192 tiles to choose from. For each small region of the image, the artist picks the tile that best matches that region. The dVAE learns both which tiles to offer (the codebook) and how to read them back into a coherent image (the decoder).

But there is a fundamental problem: picking the "best" codebook entry at each position requires an argmax operation, which is not differentiable — gradients cannot flow through a hard selection. Without gradients, we cannot train the encoder end-to-end.

The solution is the Gumbel Softmax relaxation. Instead of picking one entry, the encoder outputs a soft probability distribution over all 8,192 entries. During training, the decoder receives a weighted average of codebook vectors rather than a single vector. A temperature parameter $\tau$ controls how "soft" this averaging is: high temperature gives a smooth blend; as temperature is annealed toward zero, the distribution sharpens toward a one-hot selection — approaching argmax without ever being non-differentiable.

At inference time, the temperature is set to zero (true argmax), so each position maps to exactly one codebook entry — producing the clean, discrete tokens that the transformer will later consume.

The dVAE is trained by maximizing the Evidence Lower Bound (ELBO). The ELBO has two parts: a reconstruction term that rewards the decoder for faithfully recreating the original image, and a KL divergence term that keeps the encoder's distribution close to a uniform prior over the codebook. The balance ensures the model uses the codebook efficiently without collapsing to a few entries.

Once the dVAE is trained and frozen, every image in the dataset can be encoded as 1,024 discrete tokens. Each text caption is encoded as up to 256 BPE tokens with a vocabulary of 16,384. The text and image tokens are concatenated into a single sequence of up to 1,280 tokens, and a 12-billion-parameter decoder-only transformer is trained to model their joint distribution autoregressively.

The transformer has 64 self-attention layers, each with 62 attention heads. It uses three types of attention masks merged into a single operation. Text tokens attend to all preceding text tokens (standard causal mask). Image tokens attend to all text tokens (full cross-attention) and to preceding image tokens using a sparse attention pattern that alternates between row, column, and convolutional masks. This sparse pattern drastically reduces memory while preserving spatial coherence.

The loss function is a weighted cross-entropy: 1/8 weight on text tokens and 7/8 weight on image tokens. This reflects the priority — the model should focus on generating accurate images, using text comprehension as a means to that end.

At inference time, the model generates images by autoregressively sampling image tokens conditioned on the text prompt. But autoregressive sampling is stochastic — each run produces a different image, and quality varies. Some samples nail the prompt, others miss entirely.

The authors' solution is to generate many candidates and let a judge pick the best one. They sample 512 images for each prompt, then use CLIP — a model trained to understand text–image alignment — to score each candidate against the original text. The top-ranked images are selected as the final output.

This is like holding 512 auditions for a role and letting a casting director (CLIP) pick the best performance. The transformer is the actor; CLIP is the critic. Together, they produce results far better than any single sample.

The most remarkable property of DALL·E is its ability to compose concepts it has never seen together during training. Given "an armchair in the shape of an avocado," it generates plausible images even though no such object exists in any training set. Given "a store front that has the word 'openai' written on it," it renders readable text in context.

The model can also perform zero-shot image-to-image translation. By prefixing the prompt with "the same [object] in the style of [X]," DALL·E can change textures, materials, and artistic styles without ever being trained on paired style-transfer datasets. It handles geographic knowledge ("a photo of the food of China"), temporal knowledge ("a photo of a phone from the 1920s"), and even cross-domain analogies ("a snail made of a harp").

This zero-shot compositional ability comes entirely from scale: a sufficiently large model trained on sufficiently diverse text–image pairs learns to bind adjectives to nouns, spatial prepositions to objects, and stylistic modifiers to scenes — all through next-token prediction alone.

Let us walk through the complete DALL·E pipeline from prompt to final image:

Stage 1 (offline, once): Train the dVAE on images alone. The encoder learns to compress 256×256 images to 32×32 grids of discrete tokens (codebook size 8,192). The decoder learns to reconstruct images from these tokens. After training, the dVAE is frozen.

Stage 2 (offline, once): Encode every training image into 1,024 tokens using the frozen dVAE encoder. Encode every caption into up to 256 BPE tokens. Concatenate text + image tokens and train the 12B transformer to model the joint distribution autoregressively.

Inference: Given a new text prompt, BPE-encode it to ≤256 tokens. Sample 1,024 image tokens autoregressively from the transformer, conditioned on the text. Decode the tokens to a 256×256 image using the frozen dVAE decoder. Repeat 512 times. Use CLIP to rank all 512 candidates and return the best ones.

DALL·E's lasting contribution is not the specific architecture — autoregressive image generation has largely been replaced by diffusion. Its contribution is the paradigm: that scale and simplicity beat domain-specific engineering. A sufficiently large transformer, trained on enough text–image pairs with a standard next-token objective, can learn to compose visual concepts it has never seen together. Every text-to-image model that followed — GLIDE, DALL·E 2, Imagen, Stable Diffusion — builds on this foundation.