BLIP-2: Bootstrapping Language-Image Pre-Training with Frozen Image Encoders and Large Language Models

Vision-language pre-training (VLP) had become a victim of its own success. Models kept getting bigger — and so did the bill. Flamingo achieved strong multimodal results, but its 80 billion parameters required enormous compute to train end-to-end. Meanwhile, the ingredients for great vision-language models already existed separately: ViT encoders trained by CLIP understood images superbly, and LLMs like OPT and Flan-T5 understood language brilliantly.

The fundamental question was deceptively simple: can we just connect these existing experts instead of training everything from scratch? The challenge is that an image encoder and an LLM live in completely different representation spaces — raw visual features are meaningless to a language model. Previous attempts like Frozen used a simple image-to-text generation loss to bridge this gap, but this proved too weak. The modality gap between vision and language is deep, and a naive connection leads to catastrophic forgetting in the LLM or poor visual grounding.

The heart of BLIP-2 is the Querying Transformer (Q-Former) — a small transformer that sits between the frozen image encoder and the frozen LLM. Think of it as a translator with 32 "questions" (learnable query embeddings) that it asks the image encoder. These queries attend to the image features through cross-attention layers and learn to extract exactly the visual information that matters for language tasks.

Architecturally, Q-Former consists of two interleaved sub-modules that share the same self-attention layers: (1) an image transformer whose queries interact with frozen image features via cross-attention (inserted every other block), and (2) a text transformer that encodes or decodes text. The queries can also interact with text tokens through the shared self-attention — and different attention masks control this interaction depending on the training objective.

The Q-Former is initialized from BERT-Base weights. The cross-attention layers (which don't exist in BERT) are randomly initialized. This gives the Q-Former a strong language understanding prior from the start.

BLIP-2's training proceeds in two carefully designed stages. Each stage freezes different parts of the system and teaches Q-Former a different skill.

In the first stage, Q-Former learns to extract visual features that are most relevant to text — with the image encoder kept completely frozen. The image is passed through the frozen ViT, producing patch embeddings. The 32 learnable queries then interact with these patch embeddings through cross-attention layers, learning which visual regions matter for understanding the accompanying text.

Three objectives train Q-Former jointly in this stage, each with a different self-attention masking strategy that controls how queries and text interact:

Image-Text Contrastive Learning (ITC): Aligns the query outputs with the text embedding by maximizing similarity of matched image-text pairs and minimizing similarity of unmatched pairs. Uses a unimodal mask — queries and text cannot see each other through self-attention, forcing each modality to produce independent representations.

Image-Grounded Text Generation (ITG): Trains Q-Former to generate text conditioned on the image. Uses a causal mask — queries can attend to each other but text tokens can only see previous tokens plus all queries, mimicking a decoder that uses visual queries as a prefix.

Image-Text Matching (ITM): A binary classification task that predicts whether an image-text pair is matched. Uses a bidirectional mask — queries and text fully attend to each other, enabling fine-grained cross-modal alignment.

In the second stage, Q-Former connects to a frozen LLM. The output query embeddings from Q-Former are projected through a fully-connected layer to match the LLM's input dimension, then prepended as soft visual prompts to the LLM's input. The LLM itself remains completely frozen — it never updates a single parameter.

BLIP-2 supports two types of LLMs:

Decoder-based LLMs (e.g., OPT): Trained with standard language modeling loss — the frozen LLM generates text conditioned on the visual prompts from Q-Former.

Encoder-decoder LLMs (e.g., Flan-T5): Trained with prefix language modeling loss — the text is split into a prefix (concatenated with visual prompts as encoder input) and a suffix (used as the decoder's generation target).

This stage is what gives BLIP-2 its generative capabilities. Because the LLM is frozen, BLIP-2 inherits all the LLM's language abilities — reasoning, instruction following, knowledge — without any risk of catastrophic forgetting.

BLIP-2's results are remarkable not just for their quality, but for the efficiency with which they were achieved:

Zero-shot VQAv2: 65.0% accuracy with OPT-6.7B (65.4% with Flan-T5-XXL) — outperforming Flamingo80B's 56.3% by 8.7 points while using 54× fewer trainable parameters. Fine-tuned, BLIP-2 reaches 82.2%.

Image Captioning: Zero-shot CIDEr of 144.5 on COCO (OPT-6.7B). Fine-tuned, it achieves 145.8 CIDEr. On NoCaps, zero-shot CIDEr reaches 121.6 — a new state-of-the-art that demonstrates strong generalization to novel object categories.

Image-Text Retrieval: On COCO, fine-tuned BLIP-2 achieves 97.6% Recall@1 for image-to-text and 89.7% for text-to-image retrieval.

All this with a training cost that is a fraction of competing approaches — approximately 12 GPU-days compared to the hundreds required by end-to-end methods.

The paper's ablation study reveals critical design insights:

Representation learning is essential. Without Stage 1, both OPT and Flan-T5 give substantially lower performance on zero-shot VQA. OPT in particular suffers from catastrophic forgetting — performance drastically degrades as training proceeds. Stage 1 gives the queries enough visual grounding that the LLM can interpret them without destabilization.

All three Stage 1 losses matter. Removing ITC hurts retrieval. Removing ITG hurts generation. Removing ITM hurts fine-grained understanding. The three losses are complementary, not redundant.

Stronger unimodal models yield better results. Upgrading the image encoder from ViT-L (CLIP) to ViT-g (EVA-CLIP) consistently improves all tasks. Upgrading the LLM from OPT-2.7B to OPT-6.7B or Flan-T5-XXL improves generation quality. BLIP-2's modular design means it automatically benefits from advances in either modality.

BLIP-2's deepest legacy is the paradigm, not the architecture. Before BLIP-2, the prevailing assumption was that multimodal models needed end-to-end training of enormous systems. BLIP-2 proved that a lightweight connector trained in carefully staged phases can achieve superior results by standing on the shoulders of existing foundation models. This "freeze and bridge" pattern became the template for virtually every major multimodal system that followed — from LLaVA to GPT-4V to Gemini. The insight that modularity beats monolithic training when strong unimodal components already exist permanently reshaped how the field builds vision-language AI.