Pixel Recurrent Neural Networks

A 32×32 color image has 3,072 values (32 × 32 × 3 channels). Each pixel channel takes one of 256 integer values (0–255). The number of possible images is $256^{3072}$ — a number so vast it dwarfs the atoms in the observable universe.

A generative model must learn which of these astronomical combinations look like real photographs. Before PixelRNN, the main approaches each had a critical weakness:

• Variational Autoencoders (VAEs) provide tractable likelihood bounds but tend to produce blurry samples because the decoder averages over latent uncertainty.
• GANs produce sharp samples but give no likelihood at all — you cannot measure how "probable" an image is under the model.
• Mixture models are tractable but not expressive enough for the complexity of natural images.

The authors asked: can we model the exact joint distribution of all pixels, tractably, at scale?

The key insight is deceptively simple: use the chain rule of probability to decompose the joint distribution of all pixels into a product of conditional distributions. Flatten the 2D image into a 1D sequence by scanning left to right, top to bottom (raster scan order), then model each pixel as a conditional distribution given all previous pixels.

This turns image generation into a sequence prediction problem — exactly the kind of problem LSTMs are designed for. And crucially, this decomposition is exact: no approximation, no lower bound. The product of all the conditional probabilities is the joint probability.

Think of it like a game of dominoes: the full pattern emerges from each piece fitting against the ones already placed. Pixel 1 is unconditional — the model picks a color freely. Pixel 2 is conditioned on pixel 1. Pixel 100 is conditioned on all 99 previous pixels. By the time the model reaches the last pixel, every prior pixel has had its say.

Each pixel is not a single number but three: Red, Green, and Blue. The model decomposes each pixel's distribution further into a chain of three sub-pixel predictions:

The red channel is conditioned on all previous pixels. The green channel uses all previous pixels plus the current red value. The blue channel uses all previous pixels plus the current red and green values.

The paper introduces three architectures that all implement the same autoregressive principle but differ in how they capture the context of previous pixels:

Row LSTM processes the image row by row. For each pixel, it uses a 1D convolution along the current row (masked to see only previous pixels) as the input-to-state component, and a recurrent connection from the previous row as the state-to-state component. This gives it a triangular receptive field — it sees all pixels above but has a blind spot in the upper-right.

Diagonal BiLSTM processes the image along diagonals by skewing the input map — shifting each row by one position relative to the previous row. Two directional LSTMs scan from opposite corners, and their outputs combine. This gives the ideal receptive field: every previous pixel is visible at every layer, with no blind spots.

PixelCNN replaces recurrence entirely with stacked masked convolutions. It is significantly faster to train because all pixels can be computed in parallel, but its bounded receptive field means it captures less context than the LSTM variants.

The Row LSTM is a unidirectional layer that processes the image row by row from top to bottom. For each row, the input-to-state computation is done for all positions at once using a k×1 masked convolution — this is the parallelism trick. Then the state-to-state recurrence propagates information across the row.

Think of it as a typewriter that reads one line at a time: it can see the full page above the current line, but for the current line it types left to right, each character informed by the ones before it. The k×1 convolution lets it peek at a few pixels in the row above, giving it local spatial context.

The Diagonal BiLSTM achieves the ideal: at every layer, every pixel sees all previous pixels — no blind spots. The trick is a clever geometric transformation called skewing.

The input map is skewed by offsetting each row by one position relative to the previous row. This turns the rectangular grid into a parallelogram. Now, processing column by column is equivalent to processing the original image along its diagonals. Two directional LSTMs scan from opposite corners — one starts from the top-left, the other from the top-right — and their features are combined.

After the computation, the output is un-skewed back to the original rectangular shape. The beauty is that the 2×1 column-wise convolution in the skewed space processes one pixel from each diagonal at a time, maintaining the complete dependency field.

In a standard convolution, the kernel sees pixels in all directions — including pixels the model hasn't generated yet. This violates the autoregressive property: a pixel must only depend on previous pixels.

The solution is masking: zero out the kernel weights that would connect to the current pixel or any future pixel. Two types of masks are used:

• Mask A (applied to the first layer): blocks the center pixel and everything after it. The model cannot see its own current value.
• Mask B (applied to subsequent layers): allows the center pixel. After the first layer, features at position (i, j) already represent valid past context, so they can be included.

The mask zeroes out specific weights in the convolution filter after each gradient update, ensuring the autoregressive constraint is never violated.

PixelCNN replaces recurrence with stacked masked convolutions. Because convolutions have no sequential dependency, all pixel features within a layer can be computed in parallel — making training dramatically faster than Row LSTM or Diagonal BiLSTM.

The trade-off is the receptive field: each convolutional layer only grows the receptive field by the kernel size, so even after many layers, a PixelCNN may not see distant pixels. The LSTM variants, by contrast, can theoretically capture dependencies across the entire image through their recurrent state.

In the paper's experiments, Diagonal BiLSTM achieved the best log-likelihood (3.00 bits/dim on CIFAR-10), followed by Row LSTM (3.06), and PixelCNN came last (3.14) — precisely the order of their receptive field sizes. This confirmed that larger receptive fields directly improve generative quality.

The PixelRNN models use up to 12 LSTM layers stacked on top of each other. Training such deep recurrent networks is notoriously difficult — gradients can vanish or explode across layers. The paper adopts residual connections between LSTM layers: the input to each layer is added back to its output before passing to the next layer.

The residual block works as follows: the input map has 2h features. The LSTM layer reduces this to h features (the gate outputs). A 1×1 convolution projects the LSTM output back to 2h features, and the original input is added. This creates a shortcut path that lets gradients flow directly through the network, enabling stable training of deep models.

The Multi-Scale PixelRNN generates images in a coarse-to-fine fashion. An unconditional PixelRNN first generates a small s×s image. Then a conditional PixelRNN takes this small image as context (upsampling it via deconvolutional layers) and generates a larger n×n image.

Think of it like sketching: first you rough out the broad composition at low resolution — where the sky meets the ground, where the subject sits. Then you zoom in and fill in the details, using the rough sketch as a guide.

This approach allows the model to capture global structure at the coarse scale (where long-range dependencies span fewer pixels) and fill in local details at the fine scale.

A crucial asymmetry exists between training and generation:

During training, the target image is fully known. Because masked convolutions prevent each position from seeing future pixels, all pixel predictions can be computed in a single forward pass — full parallelism. The loss is the sum of cross-entropy losses at every pixel position.

During generation, each pixel must be sampled before the next pixel can be predicted. For a 32×32 image, this means 3,072 sequential forward passes (32 × 32 × 3 channels). This makes generation slow — but training remains efficient.

This training-generation asymmetry is a defining characteristic of autoregressive models: they are easy to train (parallel) but slow to sample from (sequential).

The paper set new state-of-the-art log-likelihood scores on every benchmark tested:

• MNIST (binary): 79.20 nats (Diagonal BiLSTM), compared to 84.55 for the previous best (DRAW).
• CIFAR-10: 3.00 bits/dim (Diagonal BiLSTM), compared to 3.24 for the previous best (NICE).
• ImageNet 32×32: First reported likelihood benchmarks on ImageNet for generative models.

The ranking across architectures was consistent: Diagonal BiLSTM > Row LSTM > PixelCNN. This directly correlated with receptive field size, confirming that capturing more context improves generation quality. The generated samples appeared crisp, varied, and globally coherent — a notable improvement over the blurriness typical of VAEs at the time.

PixelRNN demonstrated that exact likelihood and high-quality generation are not mutually exclusive. This was a paradigm shift. Before this paper, the generative modeling landscape was dominated by GANs (sharp but no likelihood) and VAEs (likelihood bound but blurry). PixelRNN carved out a third path: autoregressive models with tractable, exact likelihoods.

The ideas in this paper propagated far beyond images. The same autoregressive decomposition and masked architecture were adapted for:

• Audio: WaveNet applied the same pixel-by-pixel principle to audio waveforms, sample by sample, producing speech of unprecedented quality.
• Discrete latent codes: VQ-VAE used PixelCNN as a powerful prior over its discrete codebook, combining the strengths of autoencoders and autoregressive models.
• Language models: the raster-scan autoregressive principle is conceptually identical to left-to-right language modeling, linking this work to the GPT family.