Curiosity-Driven Exploration by Self-Supervised Prediction

Standard RL agents learn by trial and error, guided by rewards from the environment. But what happens when rewards are extremely sparse — or absent altogether?

Consider Super Mario Bros: the agent gets a score only for reaching a distant flag. With random exploration (epsilon-greedy), the probability of stumbling upon that flag through thousands of random button presses is essentially zero. The agent never gets a learning signal, so it never improves.

Humans solve this effortlessly — a child explores a playground not because anyone promised a reward, but out of sheer curiosity. This paper formalizes that intuition into a computable intrinsic reward.

A natural first idea for curiosity: predict what the next frame of pixels will look like after taking an action, and use the prediction error as reward. But this has two fatal problems.

First, predicting thousands of pixel values is extremely hard — the model wastes capacity on irrelevant details like exact textures and lighting.

Second, and more fundamentally, some parts of the environment are inherently unpredictable but irrelevant to the agent. Imagine a TV screen showing static noise in the corner of a game level. A pixel-prediction model can never predict that noise, so the agent remains permanently curious about the TV — trapped staring at randomness while ignoring the actual game.

The paper identifies three categories of environmental changes: (1) things the agent can control — like opening a door; (2) things it cannot control but that affect it — like another agent's vehicle; and (3) things it cannot control and that don't affect it — like blowing leaves or TV static. A good curiosity signal should capture (1) and (2) but ignore (3).

ICM solves the pixel trap with an elegant two-part design. Instead of predicting raw pixels, it first transforms observations into a compact feature space, then makes predictions there. The trick is how that feature space is learned.

The module has three components working together: a feature encoder $\phi$ that maps raw states to compact representations, an inverse dynamics model that learns what features matter, and a forward dynamics model whose prediction error becomes the curiosity reward.

The inverse dynamics model takes the feature representations of two consecutive states $\phi(s_t)$ and $\phi(s_{t+1})$, and predicts what action the agent took to move between them.

Why does this help? If the feature encoder includes information about blowing leaves or TV static, that information is useless for predicting the agent's action — leaves blow regardless of what the agent does. So the inverse model's training pressure forces the encoder to keep only features that are relevant to the agent's actions. This is the key insight that makes ICM robust to visual distractors.

The forward dynamics model takes the current state features $\phi(s_t)$ and the action $a_t$, and predicts what the features of the next state $\phi(s_{t+1})$ will look like.

The prediction is made in the learned feature space, not in pixel space. Since that feature space already filters out irrelevant noise (thanks to the inverse model), the forward model's prediction errors reflect genuine novelty — places the agent hasn't explored yet — rather than uncontrollable randomness.

The agent combines extrinsic reward (if any) with intrinsic curiosity reward at every time step: $r_t = r^i_t + r^e_t$. The policy is trained with A3C to maximize the expected sum of this combined reward.

All three components — encoder, inverse model, and forward model — are trained jointly with the policy. The overall loss function balances three objectives: policy learning, inverse model accuracy, and forward model accuracy.

The paper evaluates ICM in three settings, each testing a different claim about curiosity.

Sparse extrinsic reward: In VizDoom's 3D navigation maze, the agent must reach a goal from spawn points at varying distances. With dense spawning (close to goal), vanilla A3C sometimes succeeds. With sparse spawning (270+ steps away), only ICM+A3C reliably finds the goal — scoring 100% vs 0% for baseline A3C.

No extrinsic reward: With zero environmental reward, ICM-driven Mario crosses over 30% of Level-1, learning to jump over pipes, dodge enemies, and avoid pits — all discovered purely through curiosity. In VizDoom, the curious agent explores 5+ rooms while a random agent barely leaves the starting room.

Generalization: A Mario agent pre-trained with curiosity on Level-1 explores Level-2 faster than an agent trained from scratch on Level-2. The skills it learned (running, jumping, dodging) transfer to new levels. In VizDoom, a curiosity-pretrained agent fine-tuned on a new map with new textures outperforms an agent trained from scratch.

To test robustness against irrelevant stochasticity, the authors replaced 40% of the agent's visual observation with white noise — simulating the "noisy TV" problem. Results were striking: ICM with learned features succeeded just as well as without the noise. The pixel-prediction baseline (ICM-pixels) collapsed entirely.

This confirms the core thesis: the inverse-dynamics-trained feature space filters out uncontrollable factors. The agent literally cannot see the noise because its encoder learned that noise carries zero information about actions.

ICM established two principles that shaped everything that followed in exploration research.

First, predict in a learned feature space, not raw observations. This principle appears in RND (which uses a random network as the feature encoder), BYOL-Explore, and virtually every modern intrinsic motivation method.

Second, evaluate generalization. Before ICM, RL papers almost never tested on unseen environments. By showing that curiosity-learned skills transfer to new levels and maps, the paper pushed the community toward generalization as a first-class evaluation criterion.

The paper's influence extends beyond exploration: the idea that an inverse dynamics model provides a good self-supervised training signal for visual representations has been adopted in robotics, video understanding, and world models.

Today, intrinsic curiosity is a standard component in the exploration toolkit. Whether through learned features (ICM), random networks (RND), or state counting, the core idea remains: give the agent a reason to seek the unknown. ICM was the paper that made this practical at scale with raw visual observations.