Glow: Generative Flow with Invertible 1×1 Convolutions

By 2018, three families of generative models had emerged, each brilliant in one way but fundamentally limited in another:

• GANs produced the sharpest images, but they had no encoder — you couldn't map a real image to a latent code — and they offered no way to compute how likely a given image was. Training was also notoriously unstable.
• VAEs had both an encoder and a decoder, but they optimized a lower bound on the log-likelihood (the ELBO), not the likelihood itself. This approximation gap meant blurrier samples.
• Autoregressive models (like PixelCNN) computed exact likelihoods, but they generated one pixel at a time, left to right, top to bottom — making synthesis extremely slow and impossible to parallelize.

What if a model could offer all three properties simultaneously — exact likelihood, parallel synthesis, and a meaningful latent space? This is precisely where normalizing flows enter.

The mathematical foundation of all normalizing flows is the change of variables formula. The intuition: if you stretch a rubber sheet, the density of ink dots on it changes — areas that get stretched thin have fewer dots per unit area, and areas that get compressed have more. The Jacobian determinant measures exactly this stretching factor.

Start with a simple distribution you can evaluate — typically a standard Gaussian $p_Z(\mathbf{z}) = \mathcal{N}(\mathbf{0}, \mathbf{I})$. Apply an invertible transformation $f$ to produce a complex distribution over images $\mathbf{x} = f(\mathbf{z})$. The change of variables formula tells us the resulting density:

When we compose $K$ invertible steps $f = f_1 \circ f_2 \circ \dots \circ f_K$, the log-determinant simply sums: $\log\!\left|\det\!\left(\frac{\partial f^{-1}}{\partial \mathbf{x}}\right)\right| = \sum_{i=1}^{K} \log\!\left|\det\!\left(\frac{\partial f_i^{-1}}{\partial \mathbf{h}_{i-1}}\right)\right|$

The entire design challenge then becomes: how do you build invertible functions whose Jacobian determinant is cheap to compute? This is where Glow's three-part flow step comes in.

Every step of flow in Glow is a composition of exactly three invertible layers, applied in sequence. Think of it as a three-stage factory line: each stage handles a different aspect of the transformation, and because each stage is individually reversible, the whole pipeline is reversible too.

A single flow step cannot capture all the complexity of a natural image. Glow stacks $K$ flow steps into one level, then repeats for $L$ levels in a multi-scale architecture borrowed from RealNVP. At each level:

1. Squeeze: reshape each $2 \times 2 \times c$ spatial block into $1 \times 1 \times 4c$. This trades spatial resolution for channel depth, letting subsequent 1×1 convolutions mix information from neighboring pixels.
2. $K$ flow steps: apply the three-part flow step (actnorm → 1×1 conv → coupling) $K$ times.
3. Split: divide channels in half. One half goes directly to the latent space (as a factored-out Gaussian). The other half continues to the next level.

This creates a hierarchical latent space: early splits capture coarse, global features (overall shape, lighting) while later splits encode fine details (textures, edges). The factoring-out also makes training and memory more efficient — half the data exits the pipeline early at each level.

In RealNVP, the choice of which channels go to $\mathbf{x}_a$ and which go to $\mathbf{x}_b$ was fixed — either a simple reversal or a checkerboard pattern. This means the coupling layer always saw the same partition. Imagine always shuffling a deck of cards the same way: you'd never get a truly random arrangement.

Glow's invertible 1×1 convolution is a learned full-rank linear transformation of the channel dimension. It generalizes three possibilities in ascending expressiveness:

1. Reverse ordering (NICE) — fixed, zero parameters
2. Fixed random permutation (RealNVP) — fixed, zero learnable parameters
3. Learned 1×1 convolution (Glow) — $c^2$ learnable parameters

The paper shows a clear progression in log-likelihood as you move from (1) to (2) to (3), confirming that letting the model learn how to partition channels significantly improves the flow's expressiveness.

One of the most appealing aspects of Glow is how simple its training objective is. There is no adversarial game (as in GANs), no variational bound (as in VAEs), and no autoregressive decomposition. The loss is simply the negative log-likelihood averaged over the training set:

$\mathcal{L} = -\frac{1}{N} \sum_{i=1}^{N} \log p_\theta(\mathbf{x}_i)$

Each term decomposes via the change of variables formula into a Gaussian log-probability plus a sum of log-determinants — all quantities we can compute exactly.

Synthesis runs the flow in reverse: sample $\mathbf{z} \sim \mathcal{N}(\mathbf{0}, T^2\mathbf{I})$ and apply $f^{-1}$. The temperature $T$ controls quality: lower $T$ (e.g. 0.7) samples closer to the mean, producing sharper but less diverse images.

Interpolation is equally simple: encode two real images as $\mathbf{z}_A$ and $\mathbf{z}_B$, then decode $(1-t)\mathbf{z}_A + t\mathbf{z}_B$ for $t \in [0, 1]$. Because the latent space is a well-calibrated Gaussian, linear paths in latent space produce smooth, meaningful transitions in image space — smiles fade in, glasses appear, hair color changes gradually.

Because Glow's encoder maps images to a structured Gaussian latent space, we can do arithmetic with semantic meaning. For a binary attribute like "smiling":

1. Encode all smiling training images and compute their average latent vector $\mathbf{z}_{+}$.
2. Encode all non-smiling images and compute $\mathbf{z}_{-}$.
3. The attribute direction is $\Delta\mathbf{z} = \mathbf{z}_{+} - \mathbf{z}_{-}$.

To make any face smile: encode it as $\mathbf{z}$, add $\alpha \cdot \Delta\mathbf{z}$, and decode. Varying $\alpha$ smoothly controls how much the person smiles. This works because the flow's exact invertibility ensures that latent vectors near each other decode to images near each other — a property GANs cannot guarantee.

Glow achieved state-of-the-art log-likelihood on CIFAR-10 (3.35 bits/dim) and ImageNet 32×32 and 64×64 among flow-based models. But the qualitative results were arguably more striking: 256×256 face images that were, at the time, the most realistic ever produced by a non-adversarial model trained purely on maximum likelihood.

The architecture used for CelebA-HQ faces was substantial: $L = 6$ levels, $K = 32$ flow steps per level, approximately 200M parameters, and around 600 convolution layers. Training on 8 GPUs took about 2 weeks. This large capacity was necessary for modeling the long-range dependencies in high-resolution images.

Normalizing flows come with inherent constraints. Every transformation must be invertible and must have a tractable Jacobian determinant — this significantly restricts the architectural design space. The invertibility requirement means that the latent space has the same dimensionality as the data space: a 256×256×3 image maps to a latent vector of 256×256×3 entries. This is much larger than a VAE's or GAN's compact latent code and raises questions about whether the representation is truly compressed.

Computationally, Glow is expensive. Generating a single 256×256 image requires running ~600 convolution layers in reverse. The model also needed about 200M parameters to achieve good results, substantially more than contemporary GANs of similar image quality. While GANs and later diffusion models surpassed Glow in image quality, the flow paradigm's guarantee of exact likelihood remains its unique strength.

Glow demonstrated that the normalizing flow paradigm could scale to high-resolution image generation — a result that reinvigorated the entire field. Its invertible 1×1 convolution became a standard building block in virtually all subsequent flow architectures, from Flow++ to WaveGlow (for audio synthesis).

The paper also crystallized an important philosophical point: a model optimized with the simple, principled objective of maximum likelihood can produce compelling samples without adversarial training. This insight influenced the later development of continuous normalizing flows (Neural ODEs) and flow matching methods, which inherit the exact-likelihood and invertibility guarantees while using continuous-time dynamics instead of discrete steps.

Glow sits at a pivotal moment in generative modeling: the point where normalizing flows proved they could compete on image quality, not just on mathematical elegance. The specific architectural tricks — actnorm, 1×1 convolutions, multi-scale splitting — became standard vocabulary in the field. And the demonstration that maximum likelihood alone suffices for realistic synthesis planted a seed that flowered into the flow matching revolution.