Exploring Simple Siamese Representation Learning

The dream of self-supervised learning is simple: teach a network to understand images without any human labels. The strategy? Show the network two augmented views of the same image — cropped differently, color-jittered, maybe blurred — and train it to produce similar representations for both. If it can match views of the same image, it must be learning something meaningful about the image's content.

But there is a devastating shortcut. The network could learn to output the exact same vector for every input. Two views of a cat? Same vector. Two views of a car? Same vector. The similarity is perfect — and the representation is perfectly useless. This failure mode is called representation collapse, and every self-supervised method before SimSiam needed a specific mechanism to prevent it.

Before SimSiam, the field had developed four distinct anti-collapse strategies, each adding complexity. SimCLR pushed apart representations of different images (negative pairs), requiring enormous batches of 4096 or more to provide enough negatives. MoCo maintained a queue of negative representations using a momentum encoder — a slowly-updating copy of the main network. SwAV replaced negatives with online clustering using the Sinkhorn-Knopp transform. BYOL eliminated negatives entirely but relied on a momentum encoder to provide stable targets. Each method worked, but each came with significant overhead. The question SimSiam asked was radical: what if we remove all of these?

SimSiam's architecture is strikingly minimal. Start with an image x. Create two augmented views x₁ and x₂ using random cropping, color jittering, horizontal flipping, and Gaussian blurring. Both views pass through the same encoder f — a ResNet-50 backbone followed by a 3-layer projection MLP. This produces projection vectors z₁ = f(x₁) and z₂ = f(x₂).

Here is where the asymmetry enters. One branch passes through an additional prediction MLP h, producing p₁ = h(z₁). The other branch's output z₂ is treated as a fixed target via Stop Gradient. The model maximizes the cosine similarity between p₁ and stopgrad(z₂).

The full loss symmetrizes this: both branches take turns being the predictor and the target. But crucially, the Stop Gradient is always applied to the target side.

The paper's most important empirical finding is clean and dramatic. Take the SimSiam architecture. Keep everything — the encoder, the projection MLP, the prediction MLP, batch normalization, the optimizer, every hyperparameter — exactly the same. Change one thing: remove the Stop Gradient. The result? Instant collapse. The loss immediately drops to its minimum possible value of −1, the output standard deviation drops to zero (meaning all outputs are identical), and the kNN accuracy stays at chance level (0.1% on ImageNet).

Put the Stop Gradient back? The loss converges normally, the output standard deviation stays near the theoretically expected value of 1/√d (where d = 2048), the kNN accuracy steadily improves, and the final linear evaluation reaches 67.7%.

This single experiment is the paper's strongest argument. It proves that architecture alone — the predictor, batch normalization, L2 normalization — is not sufficient to prevent collapse. Stop Gradient is the essential ingredient.

The paper systematically tests every component to determine which ones are essential for preventing collapse and which merely affect accuracy.

Prediction MLP. Removing the prediction MLP h entirely causes collapse — but this is a mathematical artifact. With the symmetrized loss and no predictor, the gradient of the Stop Gradient version is identical (up to a scale of ½) to the gradient without Stop Gradient. The predictor breaks this symmetry and makes Stop Gradient meaningful. Interestingly, using a fixed learning rate (no decay) for h produces slightly better results, suggesting h should adapt to the latest representations rather than converge prematurely.

Batch size. SimSiam works remarkably well across batch sizes from 64 to 4096 — a sharp contrast to SimCLR and SwAV, which need batches of 4096 to work well. Even a batch size of 128 drops accuracy by only 0.8%. This is a major practical advantage: SimSiam is friendly to standard 8-GPU setups with a batch size of 256 or 512.

Batch normalization. Removing all BN from the MLP heads does not cause collapse (the accuracy is low at 34.6% but non-trivial). Adding BN to hidden layers helps optimization, bringing accuracy to 67.4%. Adding BN to the projection MLP's output further boosts to 68.1%. But adding BN to the prediction MLP's output causes training instability. The conclusion: BN helps optimization but does not prevent collapse.

Similarity function. Replacing cosine similarity with cross-entropy similarity still produces reasonable results (63.2% vs 68.1%). This shows collapse prevention is not tied to the specific similarity measure.

Why does Stop Gradient work? The paper proposes an elegant hypothesis: SimSiam implicitly solves an Expectation-Maximization (EM) style problem with two sets of variables.

Consider a loss function with two arguments — the network parameters θ and an auxiliary set of variables η, where ηₓ represents the "ideal" representation of image x. The optimization problem minimizes the expected distance between the network's output for augmented views and these ideal representations.

Like k-means clustering, this problem can be solved by alternating: fix η, update θ (a network training step); then fix θ, update η (assigning each image its average representation over augmentations). The Stop Gradient is a natural consequence: when solving for θ, the targets η are constants, so no gradient flows through them.

SimSiam approximates this alternation in a single step: it samples one augmentation to estimate η (instead of computing the full expectation), and takes one SGD step for θ. The predictor MLP h helps fill the gap left by this approximation — it can learn to predict the expected representation that a single sample cannot provide.

One of SimSiam's deepest contributions is conceptual: it reveals that the leading self-supervised methods are all variations of the same underlying Siamese architecture, each with one extra ingredient. SimSiam is the stripped-down core.

SimCLR = SimSiam + negative pairs. SimCLR prevents collapse by repulsing different images. Adding the predictor or Stop Gradient to SimCLR neither helps nor hurts — the contrastive mechanism already solves a different optimization problem.

BYOL = SimSiam + momentum encoder. BYOL avoids collapse using a slowly-updating copy of the encoder as the target. SimSiam shows this momentum is helpful for accuracy but not essential for preventing collapse.

SwAV = SimSiam + online clustering. SwAV replaces direct similarity with cluster assignment via the Sinkhorn-Knopp transform. Removing Stop Gradient from SwAV causes divergence — consistent with clustering being an inherently alternating formulation.

MoCo = SimSiam + momentum encoder + negative queue. MoCo combines a momentum encoder with a queue of negatives. SimSiam removes both.

On ImageNet linear evaluation with a ResNet-50 backbone, SimSiam achieves 68.1% with 100-epoch pre-training — the highest among all methods at this duration. With 200 epochs it reaches 70.0%, and with 800 epochs, 71.3%. While BYOL achieves higher accuracy at longer schedules (74.3% at 800 epochs), SimSiam accomplishes this using only a batch size of 256, standard SGD (no LARS optimizer), and no momentum encoder.

Perhaps more importantly, SimSiam's representations transfer well to downstream tasks. On PASCAL VOC detection and COCO detection and instance segmentation, SimSiam matches or exceeds the other methods, and all methods surpass or match ImageNet supervised pre-training. The "optimal" SimSiam recipe (with adjusted learning rate and weight decay) achieves the best transfer results, suggesting the learned representations capture genuinely useful visual features.

The authors emphasize a deeper point: despite the many differences between SimCLR, MoCo, BYOL, SwAV, and SimSiam, all are highly successful for transfer learning. The common factor is the Siamese architecture itself, suggesting it is the core reason for their success.

The paper closes with a thought-provoking observation. Convolutions succeeded because they encode a powerful inductive bias: translation invariance via weight sharing. A filter that detects edges works the same way everywhere in the image.

Siamese networks encode a similarly powerful bias: transformation invariance via weight sharing. By passing two augmented views through the same encoder, the network is structurally biased to produce similar outputs for inputs that differ only by augmentation — which is precisely the definition of invariance. Just as convolutions hardwire the assumption that "a pattern is the same regardless of position," Siamese networks hardwire the assumption that "an image is the same regardless of crop, color shift, or blur."

SimSiam's simplicity makes this insight sharply visible. When you strip away negatives, momentum, and clustering, what remains is a pure invariance-learning machine. The Siamese architecture is not just a convenient scaffold — it is the fundamental mechanism.