Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Before BYOL, the dominant paradigm in self-supervised visual representation learning was contrastive learning. The recipe was straightforward: take an image, create two augmented views (random crops, color jitter, blur), and train the network to make their representations similar. But this alone is not enough — the network could cheat by mapping every image to the same constant vector. To prevent this collapse, contrastive methods introduced negative pairs: representations of different images that must be pushed apart.

Think of it like a crowded conference room. Positive pairs are colleagues who should cluster together. Negative pairs are strangers who should spread out. Without the strangers, everyone would just pile into one corner.

The catch is that you need many negative pairs for this to work well. SimCLR used batch sizes of 4096 or more so that each image had thousands of negatives to push against. MoCo maintained a separate memory bank of past representations. Both approaches demanded significant computational resources, and their performance was highly sensitive to the augmentation strategy — removing color jittering from SimCLR, for example, caused accuracy to plummet by over 20 percentage points.

BYOL uses two neural networks that learn from each other: the online network and the target network. Both share the same architecture but have different weights.

The pipeline works like this: take an image and create two augmented views $v$ and $v'$. Feed $v$ to the online network and $v'$ to the target network. The online network has three stages: an encoder $f_\theta$ (e.g., ResNet-50) that produces a representation $y_\theta$, a projector $g_\theta$ (an MLP) that maps it to a projection $z_\theta$, and a predictor $q_\theta$ (another MLP) that predicts the target's projection. The target network has only the encoder $f_\xi$ and projector $g_\xi$ — no predictor. This asymmetry is crucial.

The online network is trained by gradient descent. The target network is never trained by gradients — instead, its weights $\xi$ are updated as a slow exponential moving average of the online weights $\theta$. At the end of training, the projector and predictor are discarded, and only the encoder $f_\theta$ is kept as the learned representation.

BYOL's training objective is intuitive: make the online network's prediction match the target network's projection. Both outputs are first $\ell_2$-normalized (projected onto the unit hypersphere), then compared using mean squared error — which, after normalization, is equivalent to a negative cosine similarity.

The purpose of normalization is to prevent the network from trivially solving the task by simply growing the magnitude of its outputs. By forcing everything onto the unit sphere, BYOL ensures that the network must encode directional information — the angle between vectors matters, not their length.

The loss is symmetrized: we also feed $v'$ to the online network and $v$ to the target network, computing a second loss $\tilde{\mathcal{L}}_{\theta,\xi}$. The total loss is their sum. Critically, gradients flow only through $\theta$ (the online network). The target parameters $\xi$ are updated by a Stop Gradient EMA — this is what makes BYOL fundamentally different from methods that minimize a joint loss over both networks.

The target network's weights are never updated by gradients. Instead, after each training step, they are set to an exponential moving average of the online network's weights. This is controlled by a decay rate $\tau$ that starts high (e.g., 0.996) and increases toward 1 during training following a cosine schedule.

Think of the target network as a patient mentor. It doesn't react to every new idea the student (online network) has. Instead, it slowly absorbs the student's knowledge over time, providing stable, consistent guidance. This stability is what prevents the online network from chasing its own noise and collapsing.

This is the central mystery of BYOL. Without negative pairs to push representations apart, why doesn't the network converge to a trivial constant solution — outputting the same vector for every input?

The answer lies in the interplay of two components: the asymmetric predictor and the EMA target network. Neither alone is sufficient. Remove the predictor and keep the EMA? The network collapses. Remove the EMA and keep the predictor? Also collapse. Both must work together.

The intuition comes from thinking about what the predictor is actually learning. When the predictor is near-optimal, BYOL's loss becomes the conditional variance of the target projection given the online projection. A constant (collapsed) output would maximize this variance, not minimize it — because the target would still vary while the prediction stays fixed. So the gradient pushes the online network to capture more information, not less.

The EMA target network's role is to ensure the predictor stays near-optimal. If the target changed as fast as the online network (τ = 0), the predictor would always be playing catch-up and never be close to optimal. The slow EMA gives it time to converge, maintaining the anti-collapse dynamics.

BYOL creates two augmented views of each image using a composition of standard transformations: random cropping and resizing, random horizontal flip, color jittering (brightness, contrast, saturation, hue), random grayscale conversion, Gaussian blur, and solarization. The two augmentation pipelines $\mathcal{T}$ and $\mathcal{T}'$ differ slightly in their parameter distributions.

A key finding is that BYOL is remarkably more robust to the choice of augmentations than contrastive methods. When restricted to only random crops — removing all color augmentations — BYOL's accuracy drops by 13.1 percentage points (to 59.4%), while SimCLR's drops by 27.6 points (to 40.3%). This makes sense: contrastive methods can exploit shortcuts (like matching color histograms) when those are the only distinguishing features between positive and negative pairs. BYOL, without negatives, is always incentivized to preserve all information in the target representation, not just the information that distinguishes images.

BYOL's training procedure can be summarized in five steps per iteration. First, sample an image and create two augmented views. Second, pass each view through the respective network (online gets one view, target gets the other). Third, compute the symmetrized loss — both directions. Fourth, update the online network parameters $\theta$ by gradient descent on the loss. Fifth, update the target network parameters $\xi$ using the EMA rule.

The encoder is a ResNet-50 (or larger) pretrained on ImageNet. The projector is a 2-layer MLP with batch normalization: $4096 \to 256$. The predictor has the same architecture: $4096 \to 256$. Training uses LARS optimizer with a cosine learning rate schedule, batch size of 4096, and 1000 epochs. The target decay rate follows a cosine schedule from $\tau_{\text{base}} = 0.996$ to 1.

Under linear evaluation on ImageNet (training a linear classifier on frozen representations), BYOL with a standard ResNet-50 achieved 74.3% top-1 accuracy — a 1.3% improvement over the previous self-supervised state of the art (InfoMin Aug. at 73.0%). With a ResNet-200 (2×), BYOL reached 79.6%, surpassing even the supervised ResNet-50 baseline of 76.5%.

On semi-supervised benchmarks using only 1% of ImageNet labels, BYOL outperformed SimCLR by 4.9 percentage points (53.2% vs 48.3%). On transfer learning across 12 diverse datasets — from fine-grained recognition (Birds, Cars, Aircraft) to textures (DTD) to scenes (SUN397) — BYOL outperformed SimCLR on every single benchmark and matched or exceeded the supervised ImageNet baseline on most of them.

Beyond classification, BYOL's representation transferred well to semantic segmentation (+1.1 mIoU over SimCLR on VOC), object detection (+2.3 AP50 on VOC), and depth estimation on NYU v2. This breadth of strong transfer results demonstrates that BYOL learns genuinely general-purpose visual features.

The authors ran extensive ablations to isolate what makes BYOL work. The most illuminating findings concern the decay rate $\tau$. Setting $\tau = 1$ (never updating the target) gives only 18.8% accuracy — the target is a frozen random network, so the online network can only learn limited structure. Setting $\tau = 0$ (copying the online network instantly) gives 0.3% accuracy — complete collapse, because the target changes too fast for the predictor to track. All values between 0.9 and 0.999 give strong performance (68–72%), with $\tau = 0.99$ performing best at 72.5% over 300 epochs.

Another critical ablation showed that BYOL's batch size robustness is dramatically better than SimCLR's. Reducing the batch from 4096 to 256, SimCLR's accuracy drops by about 3 points, while BYOL barely changes. This is because BYOL's loss depends only on the positive pair — the batch size only affects batch normalization statistics, not the learning signal itself.

BYOL's impact extends beyond its specific algorithm. It established that the "contrastive = negative pairs" equation is false. You can learn excellent representations by simply predicting a slowly evolving version of your own outputs. This insight simplified the self-supervised learning pipeline, reduced compute requirements, and opened the door to methods like VICReg, Barlow Twins, and the entire non-contrastive learning family. Every modern self-supervised method that avoids negative pairs traces its intellectual ancestry back to BYOL.