Dynamic Routing Between Capsules

Convolutional Neural Networks detect features using learned filters — edges, textures, parts — and that works brilliantly (as we saw with LeNet). But there is a critical weakness in how CNNs handle the arrangement of those features:

• Max pooling discards position. After detecting "eye" and "mouth," max pooling keeps only that they were found somewhere in a local region. The network cannot verify whether the eye is above the mouth — it just needs both activations to be high.

• Viewpoint changes require exponential resources. Translation invariance is built in, but rotation, scaling, and 3D viewpoint changes are not. To handle them, a CNN must either replicate filters at many orientations (exponential growth) or see every viewpoint in training data (exponential data).

• No part-whole reasoning. A CNN has no mechanism to ask: "do these detected parts agree on where the whole object should be?" It treats features as independent votes, not as a geometric puzzle.

The core insight is deceptively simple: replace individual neurons that output a single number with small groups of neurons (capsules) that output a vector.

In a CNN, a neuron says: "edge detected here, with activation 0.9." That scalar tells you how strongly a feature was detected, but nothing about its pose — its position, orientation, scale, thickness, or deformation.

A capsule says something richer:

• The length of its output vector (between 0 and 1) represents the probability that the entity exists — like the old scalar activation.

• The orientation (direction) of the vector represents the instantiation parameters — the pose: where the entity is, how big it is, how it is oriented, how it is deformed.

This is a fundamentally different kind of representation. A single capsule for the concept "eye" doesn't just fire strongly when an eye is present — its vector encodes which eye, in what position, at what angle. Think of it as upgrading from a light switch (on/off) to a compass (direction and magnitude).

Since the length of a capsule's output vector represents a probability, it must stay between 0 and 1. But we also need to preserve the vector's direction (which encodes the pose). The squashing function solves both problems: it shrinks short vectors toward zero and long vectors toward length 1, while keeping the orientation unchanged.

Think of it like a tube that compresses water flow: a trickle stays a trickle, a torrent gets capped just below maximum, but the direction of flow never changes.

This is the paper's most important contribution. How does a lower-level capsule (say, "eye") decide which higher-level capsule ("face" vs "car") to send its output to?

The answer is routing by agreement — an iterative process that works like a committee of experts finding consensus:

1. Predict. Each lower capsule $i$ makes a prediction vector for every possible parent capsule $j$ by multiplying its output $\mathbf{u}_i$ by a learned transformation matrix $W_{ij}$: $\hat{\mathbf{u}}_{j|i} = W_{ij} \mathbf{u}_i$. This is the capsule saying: "if I am part of entity $j$, then $j$ should have this pose."

2. Weight and sum. Each parent sums these predictions, weighted by coupling coefficients $c_{ij}$ (which start equal), then squashes the sum to get its output $\mathbf{v}_j$.

3. Measure agreement. If a prediction $\hat{\mathbf{u}}_{j|i}$ closely matches the parent's output $\mathbf{v}_j$ (high dot product), the coupling coefficient $c_{ij}$ increases — the part "agrees" with this whole, so send more signal there.

4. Repeat for a few iterations (typically 3). Parts that agree with a whole reinforce each other; parts that disagree get routed elsewhere.

This is like an eye capsule saying "the face should be here at this angle," a nose capsule saying "the face should be here at this angle," and if they agree — the face capsule activates strongly.

The routing algorithm runs between every pair of adjacent capsule layers. Here is the complete procedure:

• Initialize all routing logits $b_{ij} = 0$, so every lower capsule sends equal weight to all parents.
• For $r$ iterations (typically 3):
• Compute coupling coefficients $\mathbf{c}_i = \text{softmax}(\mathbf{b}_i)$
• Compute each parent's weighted input: $\mathbf{s}_j = \sum_i c_{ij} \hat{\mathbf{u}}_{j|i}$
• Apply squashing: $\mathbf{v}_j = \text{squash}(\mathbf{s}_j)$
• Update routing logits: $b_{ij} \leftarrow b_{ij} + \hat{\mathbf{u}}_{j|i} \cdot \mathbf{v}_j$

The dot product $\hat{\mathbf{u}}_{j|i} \cdot \mathbf{v}_j$ is the agreement — it measures how well capsule $i$'s prediction for parent $j$ matched what $j$ actually computed. High agreement increases the coupling; low agreement decreases it.

The paper presents a simple three-layer architecture for MNIST digit recognition:

Layer 1 — Conv1: A standard convolutional layer with 256 kernels of size 9×9 and ReLU activation. This converts pixel intensities into local feature detectors — nothing capsule-specific yet.

Layer 2 — PrimaryCapsules: A convolutional capsule layer with 32 channels of 8D capsules, using 9×9 kernels with stride 2. This produces 32 × 6 × 6 = 1,152 capsule outputs, each an 8-dimensional vector. These are the lowest-level entities: oriented edges, simple shapes. The squashing function replaces ReLU.

Layer 3 — DigitCaps: 10 capsules (one per digit class), each 16-dimensional. Every DigitCaps capsule receives input from all 1,152 PrimaryCapsules via learned transformation matrices. Dynamic routing runs between these two capsule layers.

The length of each DigitCaps vector gives the classification probability. With only 8.2M parameters (vs 35.4M for a comparable CNN baseline), CapsNet achieves 0.25% test error on MNIST.

Since each DigitCaps capsule independently represents whether a digit class is present, the paper uses a separate margin loss for each capsule. This is crucial for allowing the network to detect multiple digits simultaneously (as in the overlapping digits task).

The intuition: if digit $k$ is present ($T_k = 1$), we want its capsule's length $\|\mathbf{v}_k\|$ to be at least $m^+ = 0.9$. If digit $k$ is absent ($T_k = 0$), we want the length to be at most $m^- = 0.1$. The loss penalizes violations of these margins.

The down-weighting factor $\lambda = 0.5$ on the absent-class term prevents the network from shrinking all capsule vectors to zero at the start of training — a critical practical detail.

To verify that DigitCaps vectors actually encode meaningful pose information — not just a discriminative signal — the paper adds a reconstruction decoder: three fully-connected layers (512 → 1024 → 784) that reconstruct the input image from the capsule vector of the correct digit.

During training, all vectors except the correct digit's are masked to zero. The decoder then tries to reconstruct the original 28×28 image from just the 16D vector. The reconstruction loss (mean squared error, scaled by 0.0005) is added to the margin loss.

This serves as a powerful regularizer: it forces the capsule to encode the specific visual details of this digit — its thickness, slant, width — not just "it's a 7." When you perturb individual dimensions of the DigitCaps vector and feed them to the decoder, you can see each dimension controlling a specific visual property: stroke thickness, width, rotation, localized deformations.

The most compelling result in the paper is on MultiMNIST: two digits from different classes overlaid on the same image, with bounding boxes overlapping by about 80%.

A standard CNN struggles because max pooling merges features from both digits into an indistinguishable soup. CapsNet, using routing-by-agreement, can segment the two digits without pixel-level supervision — each DigitCaps capsule "claims" the parts that agree with its predicted pose, and the other capsule claims the rest.

The reconstruction decoder proves this works: when you reconstruct each digit separately from its capsule vector, you get clean, separated images of each digit — even though they were physically overlapping in the input. CapsNet achieves 5.2% error on MultiMNIST with just 11.36M parameters, matching a sequential attention model that was tested on a much easier (less overlapping) version of the task.

A key philosophical difference between CNNs and capsules lies in how they handle viewpoint changes:

CNNs aim for invariance: max pooling makes the output the same regardless of where a feature appears. A "7" activates the same neuron whether it's in the top-left or bottom-right. But this throws away where it is — information that matters for understanding spatial arrangements.

Capsules aim for equivariance: when the input changes (rotates, scales, shifts), the capsule's output vector changes correspondingly — in a predictable, structured way. The capsule tracks how the entity's pose changed, rather than ignoring the change.

This is why capsules can generalize to novel viewpoints that CNNs cannot. When trained only on translated MNIST digits and tested on affinely transformed digits (affNIST), an under-trained CapsNet achieved 79% accuracy compared to 66% for a CNN with similar parameters — without ever seeing rotations or shearing during training.