Contriever: Unsupervised Dense Information Retrieval with Contrastive Learning

By 2021, dense passage retrieval (DPR) had shown that neural bi-encoders can dramatically outperform keyword-based search — but only on tasks with large supervised training sets like NaturalQuestions or MS MARCO. The moment you moved to a new domain — biomedical literature, financial reports, legal documents — where no labeled query–document pairs existed, the neural retriever collapsed. Plain BM25, which simply counts how often query words appear in the document (weighted by inverse document frequency), beat every supervised dense retriever in zero-shot settings.

The core issue is that models like DPR learn their embedding space from human-labeled positive pairs: "this question goes with this passage." Without those labels, the model has no signal to organize its vector space. Meanwhile, BM25 needs no training at all — it is a fixed formula applied to raw text.

This created an uncomfortable gap: the most powerful retrieval paradigm (dense vectors + approximate nearest neighbors) was also the most brittle when labels were absent. The question Contriever asks is: can we train the embedding space itself without any labels, using only the structure of raw text?

Contriever uses the bi-encoder architecture. A single transformer encoder (BERT-base) processes queries and documents independently. Each text is fed through the encoder, and the output is computed by averaging the hidden states of the last layer — a simple mean pooling over all tokens. This produces a single dense vector for each input.

The relevance score between a query $q$ and a document $d$ is the dot product of their representations. Crucially, Contriever uses the same encoder for both queries and documents, unlike DPR which uses separate encoders. The authors found that sharing weights improves robustness in zero-shot and few-shot settings, because the model learns a single unified embedding space rather than two potentially misaligned ones.

Because documents are encoded independently of queries, the entire document collection can be pre-encoded once and stored. At query time, retrieval reduces to a fast maximum inner product search (MIPS), implemented with libraries like FAISS.

The key insight is that every document is, in some way, unique. Two excerpts from the same document should be more similar to each other than to excerpts from different documents. This is the only supervision signal Contriever needs — no human labels, no relevance judgments.

The model learns by discrimination: given a query representation $q$, it must identify the correct positive key $k^+$ (from the same document) among a pool of $K$ negative keys (from other documents). This is formalized with the InfoNCE loss, borrowed from contrastive learning in computer vision.

How you create positive pairs from a single document is the most critical design decision. Previous work used the Inverse Cloze Task (ICT): randomly select a sentence from the document as the "query," and the remaining text becomes the "key." The query and key are mutually exclusive — they have zero overlap.

Contriever instead uses independent random cropping: sample two random spans from the document, independently. The spans may overlap partially, fully, or not at all. This is the same strategy that powered SimCLR and MoCo in computer vision, adapted to text.

Why does cropping work better than ICT? Two reasons. First, partial overlap between the two views encourages the model to learn something akin to lexical matching — it discovers that shared words are a relevance signal, similar to what BM25 does naturally. Second, cropping produces symmetric distributions for queries and keys (both are random spans), which stabilizes training. ICT creates an asymmetry: the query is a single sentence while the key is an entire paragraph minus that sentence.

The authors also add random word deletion (10% of tokens) on top of cropping. This teaches the model not to rely too heavily on exact word matching and to build more robust representations.

Contrastive learning benefits enormously from more negative examples. With in-batch negatives, the number of negatives equals the batch size, so you need huge GPU memory. Contriever uses MoCo (Momentum Contrast) to decouple the number of negatives from the batch size.

MoCo maintains two networks: a query encoder $\theta_q$ updated by normal backpropagation, and a key encoder $\theta_k$ updated as an exponential moving average of the query encoder. Keys computed by the key encoder are stored in a queue. At each step, the oldest keys are dequeued and fresh keys are enqueued. This gives Contriever access to 131,072 negatives — far more than any batch size could support.

The momentum update ensures the key encoder changes slowly, so representations in the queue remain consistent. Without it, old representations would be computed by a very different version of the model, creating noisy gradients.

The headline result: on the BEIR benchmark (15 diverse retrieval datasets), the fully unsupervised Contriever outperforms BM25 on 11 out of 15 datasets for Recall@100. This is remarkable because Contriever has never seen a single labeled query–document pair — its entire training signal comes from random cropping of raw text.

On NaturalQuestions, Contriever achieves 82.1% Recall@100 versus 78.3% for BM25. On TriviaQA, both tie at 83.2%. Contriever beats all previous unsupervised dense retrievers (ICT, REALM, SimCSE) by wide margins.

When Contriever is used as pre-training before fine-tuning on MS MARCO, it achieves the best average Recall@100 (67.1) among all bi-encoder methods on BEIR, beating DPR (48.3), ANCE (60.1), TAS-B (65.0), and Splade v2 (64.8). With cross-encoder re-ranking on top, it reaches state-of-the-art on 8 of 15 BEIR datasets.

In the few-shot setting (a few hundred to a few thousand labeled examples), the contrastive pre-training gives Contriever a massive advantage: it outperforms even BERT fine-tuned on the large MS MARCO dataset as an intermediate step.

Labeled retrieval datasets are almost exclusively in English. For low-resource languages, supervised dense retrieval is not an option. This is where unsupervised pre-training shines.

The authors train mContriever, a multilingual variant initialized from mBERT and pre-trained with contrastive learning on CCNet data from 29 languages. On the Mr. TyDi benchmark (11 languages), the unsupervised mContriever already beats BM25 for Recall@100. After fine-tuning on English-only MS MARCO data, it achieves state-of-the-art Recall@100, with improvements across all languages — even ones not seen during MS MARCO fine-tuning.

Most impressively, mContriever enables cross-lingual retrieval: given a query in Arabic, Japanese, or Korean, it retrieves relevant documents from an English Wikipedia collection. This is fundamentally impossible for BM25 or any term-matching method, since the query and document scripts share no vocabulary. On the MKQA benchmark, mContriever fine-tuned on English MS MARCO outperforms the CORA retriever, which was specifically trained on cross-lingual data with translation augmentation.

The paper includes careful ablations that isolate the contribution of each design choice. The key findings are as follows.

Cropping vs ICT: Random cropping (nDCG@10 avg: 32.2) outperforms the Inverse Cloze Task (25.9) when used without fine-tuning. Adding random word deletion to cropping (33.8) further improves performance.

Number of negatives: Increasing the MoCo queue from 2,048 to 131,072 consistently improves performance, especially in the unsupervised setting. More negatives give the model harder discrimination problems, which builds better representations.

Training data: Wikipedia alone is good for Wikipedia-like tasks (FEVER), while CCNet alone excels on diverse domains (FiQA, Quora). A 50/50 mix of both gives the best overall performance.

MoCo vs in-batch negatives: The two methods perform similarly, but MoCo scales to many more negatives without requiring enormous batch sizes.

Impact of contrastive pre-training: When fine-tuned on MS MARCO, Contriever (avg nDCG@10: 46.5) significantly outperforms BERT with the same fine-tuning recipe (42.0), proving the value of the contrastive stage.