Generative Models2017intermediate10 min read

Wasserstein GAN

شبكة الخصومة التوليدية بمسافة واسرشتاين

Arjovsky, M. · Chintala, S. · Bottou, L. — ICML

The problem

Original GANs minimize the between real and generated distributions. But when these distributions live on low-dimensional manifolds — which they typically do for images — their supports rarely overlap. JS divergence is flat and maximal whenever the supports are disjoint, so the wins trivially, gradients vanish, and the learns nothing. This makes training fragile: minor changes cause (the generator memorizes a few safe outputs) or complete divergence. There is no meaningful curve to track, so debugging is guesswork.

The contribution

Replace JS divergence with the Wasserstein-1 distance (): the minimum cost of transporting mass from one distribution to another. Unlike JS, this distance varies smoothly even when distributions have disjoint supports, giving the generator useful gradients everywhere. Via the , computing this distance reduces to training a 1-Lipschitz "" — a network that scores realness without squashing through a . enforces the Lipschitz constraint. The result: stable training, no mode collapse, and a loss curve that correlates with sample quality.

The impact

WGAN reframed training as an optimal-transport problem, spawning a family of stable architectures: WGAN-GP replaced weight clipping with penalty, and Progressive GAN used WGAN's stable gradients to grow image resolution from 4×4 to 1024×1024. The idea that the loss curve should mean something — radical at the time — became standard practice. Every modern high-resolution generative model owes a theoretical debt to this paper's insight that the right distance metric matters more than network tricks.

Imagine you're a logistics manager comparing two warehouse layouts. The classic GAN approach asks a security guard: "can you tell which warehouse is which?" If the warehouses are different enough, the guard always says "yes" — but gives you zero feedback about how to rearrange boxes to fix the worse layout.

The Wasserstein GAN fires the guard and hires a moving company instead. It asks: "what is the cheapest way to move every box from Layout A to Layout B?" That cost — the Earth Mover's distance — is always informative: even when the layouts are completely different, it tells you exactly which boxes to move and how far.

The problem: when the guard is too good

In the original GAN, the discriminator classifies samples as real or fake, and the training objective is equivalent to minimizing the Jensen-Shannon divergence between the real distribution $P_r$ and the generated distribution $P_g$ .

This works fine when $P_r$ and $P_g$ overlap. But images lie on low-dimensional manifolds inside a high-dimensional pixel space — two thin sheets that almost never intersect. When the supports are disjoint, JS divergence jumps to its maximum value $\log 2$ and stays there regardless of how close the distributions are. The discriminator achieves perfect accuracy, its gradients carry no directional information, and the generator is blind.

This is the root cause of GAN training's notorious instability: mode collapse, oscillation, and the need for delicate hyperparameter tuning all trace back to a distance metric that goes blind precisely when guidance is needed most.

Open in Lab

Drag the generated distribution. JS divergence jumps to maximum the moment supports separate; Wasserstein distance changes smoothly throughout.

The demo wakes as you arrive…

The solution: pay by the shovel

The Wasserstein-1 distance — also called the Earth Mover's distance (EMD) — asks: if distribution $P_r$ is a pile of dirt and $P_g$ is the hole you want to fill, what is the cheapest way to shovel the dirt, measuring cost as (amount of dirt) × (distance moved)?

Formally, you consider every possible "transport plan" $\gamma$ that moves mass from $P_r$ to $P_g$ , and pick the one with minimum total cost. This is the optimal transport problem, studied in mathematics since Monge in 1781.

The key insight: unlike JS divergence, the Earth Mover's distance varies continuously even when distributions don't overlap. Moving $P_g$ one millimeter closer to $P_r$ always reduces the distance by a proportional amount. This continuity means gradients always carry useful information — there are no flat plateaus where the generator is stuck.

W(P_r, P_g) = \inf_{\gamma \in \Pi(P_r, P_g)} \mathbb{E}_{(x,y) \sim \gamma} [\|x - y\|]

Earth Mover's distance — the optimal transport cost — γ = a transport plan that moves mass from Pᵣ to Pᵍ · ‖x − y‖ = the distance each unit of mass travels · the infimum picks the cheapest plan among all valid ones

Open in Lab

Click "Transport!" to watch the optimal plan move mass from source to target. The total cost shown is the Earth Mover's distance.

The demo wakes as you arrive…

The trick: from transport plans to a learned critic

Computing the optimal transport plan directly is intractable for high-dimensional data. But the Kantorovich-Rubinstein duality theorem provides an alternative formulation: instead of searching over all transport plans, search over all 1-Lipschitz functions $f$ and take the supremum of the difference in expectations.

A function is 1-Lipschitz if it never changes faster than the input changes — formally, $|f(x_1) - f(x_2)| \leq \|x_1 - x_2\|$ for all $x_1, x_2$ . Think of it as a function whose slope is bounded by 1 everywhere. The value this function assigns to any point can be thought of as an "elevation score" — and the is the gap in average elevation between the real and generated distributions.

In practice, we model $f$ with a neural network — the critic. Unlike the discriminator in standard GANs, the critic outputs an unbounded real number (no sigmoid), representing how "real" an input looks. The name "critic" replaces "discriminator" to reflect this shift: it no longer classifies, it scores.

W(P_r, P_g) = \sup_{\|f\|_L \leq 1} \left[ \mathbb{E}_{x \sim P_r}[f(x)] - \mathbb{E}_{x \sim P_g}[f(x)] \right]

Kantorovich-Rubinstein duality — the critic formulation — f = a 1-Lipschitz scoring function (the critic) · the supremum finds the f that maximizes the gap between average scores on real vs generated data · that maximum gap = Wasserstein distance

Open in Lab

Drag the slope cap to see how the Lipschitz constraint limits how steeply the critic can separate real from fake scores.

The demo wakes as you arrive…

The algorithm: critic, clipping, and training loop

The WGAN training loop differs from the standard GAN in three key ways:

No sigmoid on the critic. The critic outputs a raw score, not a probability. Higher scores mean "more real." This unbounded output is what allows the Wasserstein distance to be computed.
Weight clipping. After every critic update, all weights are clamped to a range $[-c, c]$ (typically $c = 0.01$ ). This is a simple way to enforce the Lipschitz constraint: bounding the weights bounds the function's slope. It's crude but effective — the authors themselves called it "a clearly terrible way to enforce a Lipschitz constraint" while noting it already produced strong results.
More critic updates per generator step. The critic is trained $n_\text{critic}$ times (typically 5) for every generator update. Since the Wasserstein estimate improves with a better critic, investing more in the critic directly improves the gradient signal to the generator.

Open in Lab

Step through the WGAN training loop. Watch the critic train 5 times, then the generator updates once. Notice how the loss curve stays smooth.

The demo wakes as you arrive…

Critic vs Discriminator: what changed

In a standard GAN, the discriminator outputs a probability via sigmoid — "70% chance this is real." It classifies. When it's too confident, gradients vanish.

The WGAN critic outputs an unbounded score — "this image scores 3.7 on the realness scale." It evaluates quality on a continuous scale. The training signal is the difference in average scores between real and generated batches: the generator wants to increase its score, the critic wants to maximize the gap.

This unbounded scoring is exactly what makes the Wasserstein distance computable and the gradients non-vanishing. The critic can always become more precise — there's no sigmoid ceiling that kills the gradient.

Open in Lab

Left: the discriminator's sigmoid saturates and gradients flatten. Right: the critic's linear output keeps gradients flowing.

The demo wakes as you arrive…

Enforcing Lipschitz: weight clipping and its limits

The Lipschitz constraint is the price of using the Kantorovich-Rubinstein duality. The critic must not change its output too fast relative to input changes — otherwise the supremum in the duality formula would be infinite, and the Wasserstein estimate meaningless.

WGAN's solution is weight clipping: after every gradient step, force every weight $w$ into $[-c, c]$ . Since bounded weights produce bounded gradients in a feedforward network, this loosely enforces the Lipschitz constraint.

But weight clipping has clear downsides. If $c$ is too small, the critic's capacity is severely limited and gradients may vanish. If $c$ is too large, the constraint barely binds and training destabilizes. The authors acknowledged this trade-off, noting that weight clipping was chosen for simplicity rather than optimality. This limitation directly motivated WGAN-GP (gradient penalty), published later the same year, which replaced clipping with a penalty on the gradient norm — a theoretically cleaner solution.

Open in Lab

Adjust the clipping bound c. Too small: weights cluster at ±c and the critic is too simple. Too large: the Lipschitz constraint is barely enforced.

The demo wakes as you arrive…

A loss curve that means something

One of WGAN's most practical contributions is a loss curve that correlates with sample quality. In standard GANs, the generator loss oscillates meaninglessly — you can't tell from the loss alone whether the model is improving or collapsing. Researchers had to visually inspect samples, making large-scale experiments painful.

Because the WGAN critic loss approximates the Wasserstein distance, a decreasing critic loss genuinely means the generated distribution is moving closer to the real one. For the first time, GAN training had a metric you could plot, trust, and use for hyperparameter selection — just like any other task.

Open in Lab

Left: standard GAN loss oscillates without correlation to quality. Right: WGAN loss decreases smoothly and correlates with improving samples.

The demo wakes as you arrive…

Mode collapse: why WGAN resists it

Mode collapse occurs when the generator finds a few "safe" outputs that fool the discriminator and stops exploring. In a standard GAN, the JS divergence doesn't penalize missing modes — if the generator covers 3 out of 10 modes perfectly, the discriminator can't tell those 3 are fake, and the generator has no incentive to find the other 7.

The Earth Mover's distance inherently penalizes missing modes because it measures how much mass needs to be moved. If the generator piles all its probability on 3 modes, the transport cost from the remaining 7 real modes to those 3 generated modes is large. Spreading probability across all modes reduces the cost. The generator is rewarded for diversity, not just fooling the critic.

Open in Lab

Toggle between GAN and WGAN to see how the generator distributes mass across modes. The standard GAN collapses to a subset; WGAN covers them all.

The demo wakes as you arrive…

The algorithm in code

WGAN training loop — the essential differencepython

Simplified to show the idea — not the real implementation.

import torch

def train_wgan(generator, critic, data_loader, n_critic=5, clip_value=0.01):
    opt_g = torch.optim.RMSprop(generator.parameters(), lr=5e-5)
    opt_c = torch.optim.RMSprop(critic.parameters(), lr=5e-5)

    for real_batch in data_loader:
        # ── Train critic n_critic times ──────────────────────
        for _ in range(n_critic):
            z = torch.randn(real_batch.size(0), latent_dim)
            fake = generator(z).detach()

            loss_c = -(critic(real_batch).mean() - critic(fake).mean())
            opt_c.zero_grad()
            loss_c.backward()
            opt_c.step()

            # Weight clipping: enforce Lipschitz constraint
            for p in critic.parameters():
                p.data.clamp_(-clip_value, clip_value)

        # ── Train generator once ─────────────────────────────
        z = torch.randn(real_batch.size(0), latent_dim)
        fake = generator(z)
        loss_g = -critic(fake).mean()   # maximize critic score
        opt_g.zero_grad()
        loss_g.backward()
        opt_g.step()

Impact and the road ahead

2014
GAN (Goodfellow et al.)
Introduced the adversarial framework: generator vs discriminator minimizing JS divergence. Revolutionary idea, but training was notoriously unstable.
2016
DCGAN (Radford et al.)
Convolutional architecture guidelines that made GAN training more stable empirically, but didn't fix the underlying distance metric problem.
2017
WGAN (this paper)
Replaced JS divergence with Wasserstein distance. Introduced the critic, weight clipping, and meaningful loss curves. Made GAN training principled.
2017
WGAN-GP (Gulrajani et al.)
Replaced weight clipping with gradient penalty — a cleaner way to enforce the Lipschitz constraint that avoids capacity issues.
2018
Progressive GAN (Karras et al.)
Used WGAN-GP's stable training to grow images from 4×4 to 1024×1024 progressively — the first photorealistic face generator.
2019
StyleGAN (Karras et al.)
Built on Progressive GAN with style-based generation, producing the highest-quality face synthesis to date. All grounded in WGAN's training stability insights.

The story of WGAN is ultimately about asking a better question. Instead of "can you tell real from fake?" it asks "how much effort would it take to transform one into the other?" That question always has a useful answer — and useful answers produce useful gradients.

CitationArjovsky, Chintala, Bottou. Wasserstein GAN. ICML, 2017.

Terms in this paper

Wasserstein Distanceمسافة واسرشتاين
Earth Mover's Distanceمسافة ناقل الأرض
Criticالناقد
Lipschitz Continuityاستمرارية ليبشتز
Weight Clippingقصّ الأوزان
Mode Collapseانهيار الأنماط
Jensen-Shannon Divergenceتباعد جنسن-شانون
Kantorovich-Rubinstein Dualityثنائية كانتوروفيتش-روبنشتاين
Training Stabilityاستقرار التدريب
Generative Adversarial Network (GAN)الشبكات التوليدية التنافسية