Flamingo: a Visual Language Model for Few-Shot Learning

By 2022, the AI community had two kinds of powerful models that rarely talked to each other. On one side, CLIP and similar contrastive models could match images to text — but only via a shared embedding space; they couldn't generate language. On the other side, large language models like Chinchilla could write fluent paragraphs — but were completely blind, processing only text tokens.

The obvious fix — fine-tune the LM on visual data — ran into a brutal obstacle: catastrophic forgetting. When you retrain a 70-billion-parameter language model on captioning data, it can lose its ability to do language tasks. And even if you manage the forgetting, you still need large labeled datasets for every new visual task. Want to do VQA? Train on VQA data. Want captioning? Train on captioning data. There was no visual equivalent of GPT-3's in-context learning, where you show a few examples in the prompt and the model generalizes on the spot.

Flamingo's key insight is architectural restraint. Instead of merging vision and language into one model, it keeps both the vision encoder and the language model frozen and builds a lightweight bridge between them. This bridge has two components:

1. Perceiver Resampler — compresses variable-length visual features from images or video frames into a fixed set of visual tokens.
2. Gated Cross-Attention Dense (GATED XATTN-DENSE) layers — new layers inserted between the frozen LM layers that allow the language model to "look at" the visual tokens through cross-attention.

The frozen components (vision encoder + LM) hold 90%+ of the parameters and never change. Only the bridge — the Perceiver Resampler and the gated cross-attention layers — is trained from scratch. This preserves the LM's language capabilities while teaching it to incorporate visual information.

A single image produces hundreds of visual features from the vision encoder (e.g. a 2D spatial grid from an NFNet). A video produces thousands — one grid per frame. Feeding all of these directly into the language model would be prohibitively expensive, especially since the LM's attention is quadratic in sequence length.

The Perceiver Resampler solves this with an elegant idea borrowed from Perceiver: use a fixed set of learned latent queries (64 tokens) that cross-attend to the visual features. No matter whether the input is one image or a 30-frame video, the output is always exactly 64 visual tokens. Think of it as a panel of 64 expert analysts: they each look at the entire visual input and produce one compact summary vector. The same 64 analysts handle any input, so the downstream LM always sees a fixed-size visual context.

Architecturally, the Perceiver Resampler is a small Transformer with learned latent vectors as queries and the vision encoder's features (plus learned temporal embeddings for video) as keys and values. It has 194M parameters — tiny compared to the 70B LM.

The Perceiver Resampler produces 64 visual tokens per image. But how does the language model actually use them? This is where Gated Cross-Attention Dense (GATED XATTN-DENSE) layers come in.

New cross-attention layers are inserted between the existing frozen LM layers. In each GATED XATTN-DENSE layer, the query comes from the language model's hidden states (what the LM is currently "thinking"), while the key and value come from the visual tokens produced by the Perceiver Resampler. This lets each text token ask: "Is there anything in the image relevant to what I'm about to generate?"

But there's a critical subtlety. If you simply add cross-attention outputs to the LM's hidden states, you will corrupt the carefully learned language representations at initialization, crashing performance. Flamingo's solution: tanh gating. The output of each new layer is multiplied by $\tanh(\alpha)$, where $\alpha$ is a learnable scalar initialized to zero. At initialization, $\tanh(0) = 0$, so the visual signal is completely muted — the model behaves exactly like the original frozen LM. During training, $\alpha$ gradually increases, letting visual information flow in smoothly. This is the "gate" that opens slowly, preventing the language model from being overwhelmed.

Most vision-language models of the era handled one image and one text at a time. Flamingo breaks this limitation by accepting arbitrarily interleaved sequences of images (or videos) and text, exactly as they appear on a web page: an image, then a paragraph, then another image, then another paragraph.

Special tokens mark the boundaries: <image> precedes each visual input, and <EOC> (End of Chunk) follows each text segment associated with an image. This structure is critical for few-shot learning: you can place several (image, caption) pairs in the prompt as demonstrations, then provide a new image and let the model generate the caption.

To prevent future images from influencing past text (causal consistency), Flamingo uses image-causal masking in the cross-attention: each text token can only attend to visual tokens from the image that precedes it, not from images that come later in the sequence.

Flamingo is trained on four complementary datasets:

• M3W (MultiModal MassiveWeb): text and images extracted from approximately 43 million web pages, preserving their natural interleaved structure. This is the key dataset for in-context few-shot learning.
• ALIGN: 1.8 billion image-text pairs from alt-text on the web. Large but noisy, with short descriptions averaging 12.4 tokens.
• LTIP (Long Text & Image Pairs): 312 million image-text pairs with higher quality and longer descriptions (averaging 20.5 tokens).
• VTP (Video & Text Pairs): 27 million short videos (~22 seconds) paired with descriptive captions.

The training objective is a weighted sum of per-dataset negative log-likelihoods, with weights $\lambda$ set to 1.0, 0.2, 0.2, and 0.03 for M3W, ALIGN, LTIP, and VTP respectively. The M3W dataset receives the highest weight because it provides the interleaved structure essential for few-shot capabilities. Ablation studies showed that removing M3W caused a 17% performance drop — confirming that interleaved data is the backbone of few-shot learning.

The whole architecture comes together in few-shot evaluation. Given a new visual task — say, answering questions about medical images — you don't fine-tune anything. You simply construct a prompt with a few (image, question, answer) examples, append a new (image, question), and let Flamingo generate the answer. The interleaved architecture, trained on M3W's natural web structure, has learned the meta-skill of in-context learning from visual demonstrations.

Flamingo supports 0-shot (no examples), 4-shot, 8-shot, 16-shot, and 32-shot evaluation. Performance improves log-linearly with the number of shots, and even at 4 shots, Flamingo often matches or exceeds models that were fine-tuned on tens of thousands of labeled examples.

Flamingo comes in three sizes, all sharing the same 435M-parameter vision encoder and 194M-parameter Perceiver Resampler:

• Flamingo-3B: 1.4B frozen LM + 1.2B trainable GATED XATTN-DENSE (inserted after every LM layer). 3.2B total parameters.
• Flamingo-9B: 7.1B frozen LM + 1.6B trainable GATED XATTN-DENSE (inserted every 4th layer). 9.3B total parameters.
• Flamingo-80B: 70B frozen Chinchilla LM + 10B trainable GATED XATTN-DENSE (inserted every 7th layer). 80B total parameters. Trained on 1536 TPU chips for 15 days.

A key design decision: larger models insert cross-attention less frequently. The 80B model only adds a gated cross-attention layer every 7th frozen LM layer, trading off visual resolution for computational efficiency.

Flamingo-80B was evaluated on 16 multimodal benchmarks spanning captioning, visual question answering, and classification. The results were striking:

• With just 32 few-shot examples, Flamingo set new state-of-the-art on 6 out of 16 benchmarks — without any fine-tuning — surpassing models that were fine-tuned on thousands of labeled examples.
• On all benchmarks with published few-shot results, Flamingo set the new state-of-the-art for few-shot performance.
• Performance scaled cleanly with model size: Flamingo-80B > Flamingo-9B > Flamingo-3B across all tasks.
• On open-ended tasks like COCO captioning and VQAv2, Flamingo showed especially strong performance, demonstrating that the generative language model backbone was a major advantage.

When fine-tuned on task-specific data, Flamingo set new state-of-the-art on an additional 5 benchmarks, demonstrating that the few-shot architecture is also an excellent initialization for full fine-tuning.

Flamingo's deepest contribution is the "freeze-and-bridge" paradigm. Before Flamingo, multimodal models either trained everything end-to-end (expensive and prone to forgetting) or used simple projection layers (losing rich visual information). Flamingo showed that you can keep powerful pretrained models intact and connect them with carefully gated bridges. This pattern — frozen encoder, bridge module, frozen decoder — is now the standard recipe for building multimodal models, from BLIP-2 to LLaVA to PaLM-E.