Representation Learning with Contrastive Predictive Coding

Before CPC, the dominant approach to unsupervised learning was generative modeling: train a model to reconstruct its input (like a deep autoencoder) or predict every pixel or sample. The problem is that most of the information in raw signals — the exact brightness of each pixel, the precise waveform of audio — is high-frequency noise that says nothing about the underlying meaning.

A model that tries to perfectly reconstruct a face image spends most of its capacity modeling skin texture and lighting, not learning that it's a face. A speech model that reconstructs waveforms spends capacity on room acoustics, not phonemes.

What we actually want is the high-level, slow-changing structure: the melody across an audio clip, the object identity across image patches, the topic across a paragraph. That structure is what makes downstream tasks — classification, retrieval, reasoning — possible.

CPC's insight is a shift in what you predict and how you evaluate the prediction:

What: Don't predict raw pixels or waveforms. Instead, predict a compact representation of the future — an embedding that captures the high-level content while discarding noise.

How: Don't compare your prediction to the truth pixel-by-pixel. Instead, treat it as a classification problem: can you pick the true future embedding out of a lineup of random negatives?

This combination is powerful. The model is forced to capture the shared, slow-varying structure between past and future (maximizing mutual information) while ignoring unpredictable noise. And since the prediction happens in embedding space, it's fast and works for any modality — audio, images, text, RL.

CPC chains three components end-to-end — think of them as a microphone, a narrator, and a detective:

1. Encoder $g_{\text{enc}}$ — the microphone. It compresses each raw observation $x_t$ (an audio frame, an image patch, a word) into a latent embedding $z_t = g_{\text{enc}}(x_t)$. This strips away high-frequency noise and keeps the useful content. For speech, the encoder is a strided convolutional network running directly on raw 16 kHz audio; for images, it's a ResNet; for text, a word embedding lookup.

2. Autoregressive model $g_{\text{ar}}$ — the narrator. It reads the sequence of embeddings $z_1, z_2, \ldots, z_t$ up to the current time step and produces a single context vector $c_t = g_{\text{ar}}(z_{\leq t})$ that summarizes everything seen so far. This is a GRU, LSTM, or masked CNN — any model that captures temporal order. The context is the model's "running story" of what has happened.

3. Linear predictors $W_k$ — the detectives. For each future step $k$ (1, 2, 3, …), a separate linear transform $\hat{z}_{t+k} = W_k c_t$ predicts what the future embedding should look like. Each predictor specializes in a different "look-ahead distance."

The key question is: how do you train this pipeline without labels? The answer is a cleverly designed loss function called InfoNCE (Information Noise-Contrastive Estimation).

The setup is like a police lineup: at every time step $t$ and for each future step $k$, assemble a set of $N$ embeddings. Exactly one is the real future $z_{t+k}$ (the positive); the other $N-1$ are random embeddings drawn from the data (the negatives). The model must identify which one is real, using only its context vector $c_t$.

The scoring function is a simple dot product that measures compatibility: $f_k(x_{t+k}, c_t) = \exp(z_{t+k}^\top W_k c_t)$. Higher score means "this looks like the real future given my context."

Read it as a softmax classifier over candidates: the loss is categorical cross-entropy where the "correct class" is the true future. If the model perfectly distinguishes the positive from the negatives, the loss reaches its minimum $-\log(1) = 0$. If it guesses randomly, the loss is $\log(N)$.

The beautiful part is the theoretical guarantee: minimizing InfoNCE is equivalent to maximizing a lower bound on mutual information between $c_t$ and $x_{t+k}$: $I(x_{t+k}; c_t) \geq \log(N) - \mathcal{L}_N$. As you add more negatives, the bound gets tighter — the model is forced to capture more shared information.

Mutual information $I(x_{t+k}; c_t)$ measures how much knowing the context $c_t$ reduces your uncertainty about the future $x_{t+k}$. Maximizing it forces the model to encode exactly the information that is shared between past and future — the slow, structural patterns — while ignoring the information that is private to each timestep — the noise.

Think of it as a filter for relevance: the melody is shared between the first and second bar of music (high mutual information), but the exact recording noise at each moment is independent (low mutual information). The model learns to represent the melody and discard the noise — not because we told it to, but because that's the only way to solve the contrastive task.

This is why CPC captures general-purpose features: the shared structure across time is exactly what downstream tasks need — phoneme identity in speech, object identity in images, topic coherence in text.

A critical design choice: CPC does not model the full conditional distribution $p(x_{t+k}|c_t)$ or the marginal $p(x_{t+k})$. It only models their ratio:

$f_k(x_{t+k}, c_t) \propto \frac{p(x_{t+k}|c_t)}{p(x_{t+k})}$

This is the density ratio: it tells you how much more likely a future observation is given the context versus in general. CPC learns this ratio directly via the contrastive loss, sidestepping the need for any generative model.

Why is this clever? Modeling $p(x)$ in high-dimensional spaces (pixels, audio) is extremely hard and wastes capacity on noise. But the ratio $p(x|c)/p(x)$ captures only the dependency between context and future — exactly the mutual information we want. It's the shortcut from raw data to high-level understanding.

The paper's strongest claim is universality: the same contrastive-predictive objective works across fundamentally different data types. The encoder changes, but the loss and the architecture pattern remain identical:

Audio (LibriSpeech): The encoder is a 5-layer strided CNN on raw 16 kHz waveforms, producing a feature vector every 10ms. The autoregressive model is a GRU. Despite never seeing phone labels, the representations achieve 64.6% accuracy on phoneme classification with a simple linear probe — matching supervised baselines.

Vision (ImageNet): Images are split into a 7×7 grid of overlapping patches. Each patch is encoded by a ResNet-v2-101. A masked CNN (like PixelCNN) serves as the autoregressive model, predicting downward column by column. CPC achieves 48.7% top-1 accuracy with a linear classifier on the frozen representations.

Text (BookCorpus): Each sentence is the unit. A 1D CNN encodes words into a sentence embedding; a GRU summarizes past sentences into context. The model predicts future sentence embeddings.

Reinforcement Learning (DeepMind Lab): CPC is added as an auxiliary loss to an A3C agent in 3D navigation tasks. It improves sample efficiency and speeds up learning, especially in visually complex environments.

The gold standard for evaluating unsupervised representations is the linear probe: freeze the learned encoder, then train only a linear classifier on top. If a simple linear layer can classify well, the representations have already organized the data into linearly separable clusters — proving the encoder learned meaningful structure, not just memorized patterns.

On speech phoneme classification with LibriSpeech, CPC representations achieve 64.6% phone accuracy — competitive with fully supervised approaches and far above random or MFCC baselines. On ImageNet with a linear classifier on frozen ResNet-v2 features, CPC reaches 48.7% top-1 accuracy. On the text domain, sentence-level representations show strong performance on sentiment classification and question-type detection.

Perhaps most revealing: CPC representations encode multiple levels of structure simultaneously. In the speech domain, the same representation that predicts future audio also captures both phoneme identity and speaker identity — two orthogonal tasks — in linearly separable subspaces. The model wasn't trained on either task, yet both are readable from the same embedding.

CPC is the bridge between Word2Vec's intuition — "predict your neighbors to learn meaning" — and the modern self-supervised paradigm where a single pre-trained encoder, trained without labels, powers everything from speech recognition to image search to multimodal understanding.