ALIGN: Scaling Up Visual and Vision-Language Representation Learning with Noisy Text Supervision

By 2021 the recipe for visual representation learning was clear: pre-train on a large labeled dataset, then fine-tune on downstream tasks. The problem was that "large labeled dataset" meant either ImageNet (14M images with manually verified class labels) or JFT-300M (Google's internal dataset with 300M labeled images). Both required years of human effort.

For vision-language tasks the situation was worse. Conceptual Captions, the standard dataset, started with billions of raw image-caption pairs from the web — then applied heavy semantic parsing, text cleaning, image-based filtering, and manual quality checks to produce a mere 3.3 million clean pairs. This process threw away 99.7% of the data. The quality was high, but the scale was tiny compared to what NLP models were consuming.

ALIGN asks a radical question: what if we skip almost all the cleaning and just use the noisy data? The hypothesis is that at sufficient scale, the signal in billions of imperfect pairs drowns out the noise.

ALIGN's dataset starts from raw English alt-text data — the text that web developers write in the alt attribute of <img> tags. This text is often noisy: generic strings like "thumbnail", file names like "IMG_2847.jpg", or descriptions of the frame rather than the content. The paper applies only minimal filtering: remove images smaller than 200 pixels, exclude alt-texts shared by more than 10 images, discard texts with rare tokens, and filter texts shorter than 3 words or longer than 20. No semantic parsing, no human review. The result: 1.8 billion image-text pairs — nearly 600× the size of Conceptual Captions.

ALIGN uses a dual-encoder architecture — deliberately simple. There are two independent encoders that never look at each other's inputs:

Image encoder: EfficientNet (up to the L2 variant) with global average pooling. The pooled features pass through a linear projection to produce a fixed-size embedding vector. No region features, no object detectors, no attention over patches — just the whole image compressed into a single vector.

Text encoder: BERT (up to Large) with a 100K wordpiece vocabulary generated from the training data. The [CLS] token embedding passes through a linear layer to match the image embedding dimension.

Both embeddings are L2-normalized and live in a shared space. Matching is done by cosine similarity. This architecture is orders of magnitude faster at retrieval than cross-attention models like UNITER or ViLBERT, because each image and text can be encoded independently and compared via a simple dot product.

The training objective is elegant in its simplicity. Given a batch of N image-text pairs, ALIGN treats each matched pair as a positive example and all other N−1 pairings as negatives. Two symmetric losses are computed:

Image-to-text: for each image, classify which of the N texts is its match. Text-to-image: for each text, classify which of the N images is its match.

Think of it as a conference room with N people, each holding a photo and a caption. Everyone's task is to find whose caption matches their photo, and vice versa. The more people in the room (larger batch size), the harder the task and the better the learned representations.

A critical training detail is the batch size. ALIGN uses 16,384 pairs per batch, spread across 1,024 TPUv3 cores (16 pairs per core). Embeddings from all cores are concatenated to form the full batch for the contrastive loss. This massive batch means each image is compared against 16,383 negative texts — a much harder classification task that produces better embeddings.

The learned temperature σ converges to approximately 1/64. A smaller σ sharpens the softmax distribution, making the model more discriminative. The paper also applies label smoothing (0.1) to the softmax losses, which prevents the model from being overconfident and helps with noisy labels.

The ablation study reveals clean scaling laws. The paper sweeps image encoders from EfficientNet-B1 to L2 and text encoders from BERT-Mini to BERT-Large. Results improve smoothly with both axes — no diminishing returns, no collapse.

For vision-only tasks like ImageNet KNN, the image encoder size dominates: even with BERT-Mini as the text encoder, EfficientNet-L2 outperforms B7 with BERT-Large. For cross-modal tasks like retrieval, both encoders matter equally. This makes intuitive sense: retrieving the right caption requires understanding both the image content and the linguistic nuance.

The embedding dimension also matters. Going from 80 to 640 dimensions steadily improves performance, so the final model lets the dimension scale with the backbone (L2 uses 1,376-dimensional embeddings). The number of in-batch negatives is equally important — using only 25% of the batch as negatives degrades retrieval R@1 by about 3 points.

Once the shared embedding space is learned, zero-shot classification becomes trivial. To classify an image into one of K classes, feed each class name through the text encoder (e.g. "A photo of a dog"), compute cosine similarity with the image embedding, and pick the highest-scoring class. No training on the target task at all.

ALIGN achieves 76.4% top-1 zero-shot accuracy on ImageNet — comparable to CLIP's 76.2% despite using noisier data. On ImageNet-R (non-natural images like cartoons and sketches), ALIGN reaches 92.2% vs CLIP's 88.9%, showing that noisy training data may actually improve robustness to distribution shifts.

For retrieval, ALIGN sets new records on both Flickr30K and MSCOCO, outperforming even cross-attention models that are orders of magnitude slower. With fine-tuning, ALIGN reaches 95.3% R@1 on Flickr30K image-to-text retrieval — a 7+ point jump over the previous best.

One of the most striking properties of ALIGN's embedding space is compositionality. Because images and text live in the same vector space, you can add or subtract text embeddings from image embeddings to modify the query semantically.

Want to find images of "pandas in Australia"? Take the embedding of a panda image, add the text embedding of "Australia", and retrieve the nearest images. The results show koala-like animals in Australian landscapes — the embedding space has learned to compose visual and linguistic concepts algebraically, much like the famous word2vec analogies ("king − man + woman = queen") but across modalities.

You can also subtract concepts: take a street scene, subtract "cars", and the retrieval returns similar streets but without vehicles. This emergent capability was not explicitly trained — it falls out naturally from the contrastive objective applied at massive scale.

Because ALIGN's data pipeline uses no language-specific filters beyond basic text processing, extending it to other languages is straightforward. The paper lifts the English-only constraint, collects 1.8B multilingual image-text pairs (covering 100+ languages), builds a 250K wordpiece vocabulary, and trains ALIGN_mling using the same architecture and hyperparameters.

On Multi30K (multilingual image-text retrieval covering English, German, French, and Czech), zero-shot ALIGN_mling outperforms the fine-tuned M3P model on English, German, and French. The largest improvement is +57.8 mean recall on French — a language where specialized multilingual models had struggled. This demonstrates that ALIGN's scale-over-curation philosophy generalizes across languages with minimal engineering effort.

ALIGN is trained from scratch (no pre-trained weights for either encoder) using the LAMB optimizer with a weight decay of 1e−5. The learning rate warms up linearly to 1e−3 over 10K steps, then decays linearly to zero over 1.2M steps (~12 epochs). Images are resized to 346×346 and randomly cropped to 289×289 during training (center-cropped during evaluation).

The text encoder uses a maximum sequence length of 64 wordpiece tokens, since alt-texts are capped at 20 unigrams. The softmax temperature is initialized at 1.0 and learned during training. Label smoothing of 0.1 is applied to both contrastive losses.

For fine-tuning on retrieval benchmarks, the batch size is reduced from 16,384 to 2,048 to avoid false negatives (when the dataset is small, random pairs may actually be true matches). The learning rate drops to 1e−5, and the model is fine-tuned for 3K–6K steps depending on the dataset.

ALIGN's lasting contribution is not just a model but a principle: you don't need to clean your data if you have enough of it. This philosophy has reshaped how the field collects training data. Before ALIGN, data curation was considered a prerequisite for quality. After ALIGN, researchers increasingly embraced web-scale noisy data as a viable — sometimes superior — alternative. The trend continues today in large multimodal models that train on billions of uncurated image-text pairs from the open web.