Mastering Diverse Domains Through World Models

DreamerV3 operates in three phases that repeat in a loop. First, the agent interacts with the real environment and stores experiences in a replay buffer — raw observations, actions, and rewards. Second, the world model learns to compress these experiences into a compact latent space and predict future states and rewards. Third, an actor-critic pair improves behavior by rehearsing entirely inside the world model's imagination, never touching the real environment again.

The key insight is that phase three — the imagination phase — is vastly cheaper than real interaction. The agent can simulate thousands of future trajectories in the time it would take to run a single real episode. This makes DreamerV3 extraordinarily sample-efficient: it extracts maximum learning from minimal real experience.

The world model is built on the Recurrent State-Space Model (RSSM), which combines two complementary representations. A deterministic path uses a GRU recurrent network to maintain a persistent memory of past events — think of it as a running summary of everything that has happened. A stochastic path captures the inherent uncertainty in what happens next — because the future is never fully predictable.

Together, these form a compact latent state at each timestep. The deterministic state $h_t$ is the GRU's hidden state; the stochastic state $z_t$ is a collection of 32 categorical distributions, each with 32 classes, sampled as one-hot vectors. This discrete representation was shown in DreamerV2 to be more expressive than Gaussian latents for modeling complex environments.

The model has four components working in concert: an encoder compresses raw observations into the stochastic state; a dynamics predictor forecasts the next stochastic state from the deterministic state alone (enabling imagination without observations); a reward predictor estimates the reward at each state; and a decoder reconstructs observations to keep the latent space grounded in reality.

Different environments produce wildly different numbers. A robotics task might give rewards between -1 and 1. A video game might give rewards in the thousands. Pixel values range from 0 to 255. If the neural network must handle all these scales with the same weights, training becomes unstable — large values dominate gradients and small values get ignored.

Previous solutions included reward clipping (which destroys information) or running statistics normalization (which is non-stationary and fragile). DreamerV3 introduces a simpler approach: the symlog transform.

DreamerV3 applies symlog in three places: the encoder squashes input observations, the reward predictor outputs symlog-space predictions decoded via a twohot distribution, and the critic also predicts values in symlog space. The twohot encoding is a lightweight form of distributional reinforcement learning — instead of predicting a single value, the network predicts a categorical distribution over a discretized range of values, then computes the expected value. This provides richer gradient signals than point estimates.

The world model must balance two competing pressures. The posterior (encoder) tries to pack as much observation information as possible into the latent state. The prior (dynamics predictor) tries to predict the next latent state from memory alone, without seeing the observation. The KL divergence between them measures how much the encoder is smuggling in information that the dynamics can't predict.

If the KL penalty is too strong, the encoder gives up and ignores observations — the latent states become trivially predictable but useless. If it's too weak, the dynamics predictor doesn't learn because the encoder is doing all the work. Previous approaches required careful per-domain tuning of this balance.

DreamerV3 combines two techniques to make this robust. KL balancing uses an 80/20 split: 80% of the gradient trains the dynamics predictor to match the posterior (forward KL), while 20% trains the encoder to stay close to the prior (reverse KL). This asymmetry biases toward training the dynamics predictor, which is the component used during imagination. Free bits clamp the KL below 1 nat, allowing the encoder to encode at least 1 nat of information per timestep without penalty. This prevents the degenerate solution of an empty latent space.

Two more robustness techniques complete DreamerV3's domain-agnostic design.

Unimix categoricals. In the RSSM and the actor network, categorical distributions are mixed with 1% uniform noise: $p_{\text{mix}} = 0.99 \cdot p_{\text{net}} + 0.01 \cdot \text{Uniform}$. This prevents any action or latent class from having zero probability. Even a well-trained policy always retains a 1% chance of trying something unexpected. Think of it as a fire exit that is always unlocked — the agent can never get permanently stuck in a behavioral dead-end.

Percentile return normalization. Traditional return normalization divides by the running standard deviation, but when rewards are sparse and the standard deviation is near zero, this amplifies noise catastrophically. DreamerV3 instead tracks the 5th and 95th percentiles of returns using exponential moving averages, then scales returns by the gap between them (only when the gap exceeds 1). This makes the actor insensitive to the absolute scale of rewards — a reward of 0.01 in one domain and 10,000 in another produce comparable gradient magnitudes.

Once the world model is trained, DreamerV3 improves its policy entirely inside the model's imagination. Starting from states sampled from the replay buffer, the agent rolls out imagined trajectories using the dynamics predictor — no real environment interaction is needed. The actor (policy network) selects actions, and the critic (value network) estimates the expected cumulative reward from each state.

The actor maximizes the critic's value estimates through straight-through gradients: the policy gradient backpropagates through the entire imagined trajectory. An entropy regularizer with a fixed scale encourages exploration, and its effectiveness is guaranteed by the percentile normalization — the actor sees similarly scaled returns regardless of the domain.

The critic is trained with symlog twohot predictions and regularized toward an exponential moving average (EMA) of its own past outputs, smoothing training dynamics. A slow target network provides stable bootstrap targets, preventing the value function from chasing its own rapidly changing estimates.

DreamerV3 was evaluated across seven benchmark suites with a single set of hyperparameters: Proprio Control (continuous), Visual Control (pixels), Atari 200M, BSuite, Crafter, DMLab, and Minecraft. In every suite, it matched or outperformed algorithms that were specifically tuned for that domain.

The most dramatic result was in Minecraft. Collecting a diamond requires navigating an open 3D world from pixel observations, crafting a technology tree of 17 items in the correct order, and doing so with extremely sparse rewards (the only reward signal is obtaining the diamond). Previous attempts either used human demonstrations, curriculum learning, or massive compute (VPT used 720 V100 GPUs for 9 days with contractor data). DreamerV3 collected diamonds end-to-end, from scratch, using a single V100 GPU — the first algorithm to do so.

The authors also demonstrated favorable scaling properties: larger models directly translated to higher data efficiency and final performance, with no instability or diminishing returns within the tested range.

Beyond the robustness techniques, DreamerV3 introduces several architectural refinements. The GRU is replaced by a Block GRU that applies the gating mechanism in parallel across blocks rather than element-wise, improving computational efficiency. All layers use LayerNorm (specifically RMSNorm in later versions) followed by SiLU activations, replacing the earlier use of ELU. The optimizer uses adaptive gradient clipping to stabilize training, and the replay buffer includes an online queue for recent experiences alongside the larger uniform-sampling buffer.

These may seem like minor engineering choices, but collectively they enable DreamerV3 to train stably with large networks. The robustness of training with LayerNorm and SiLU was critical for scaling — earlier versions of Dreamer could not use models as large without diverging.