Diffusion Policy: Visuomotor Policy Learning via Action Diffusion

Teaching a robot to perform a task from human demonstrations seems like a straightforward supervised learning problem: observe what the human does and learn a mapping from observations to actions. In practice, three properties make this far more challenging than typical regression.

First, human demonstrations are multimodal. Given the same observation — say, a block sitting in front of a robot arm — there may be equally valid actions: push it left, push it right, or go around it. A policy that averages these modes produces an action that goes straight through the block — a move that belongs to none of the valid strategies.

Second, robot actions are sequential and correlated. A single action in isolation is meaningless; what matters is a coherent trajectory. Predicting one step at a time invites jitter — the policy may switch between valid strategies on consecutive steps, producing incoherent motion.

Third, the action space can be high-dimensional. A bimanual robot with grippers has 14 or more degrees of freedom per timestep, and predicting a sequence of such actions compounds the dimensionality.

Diffusion models were originally designed to generate images. The key insight of this paper is that the same mechanism — start with noise and iteratively denoise — works remarkably well for generating robot actions.

In image diffusion (DDPM), a neural network learns to predict the noise added to an image, then removes it step by step. In Diffusion Policy, the same process happens in action space: the model starts with a random action sequence sampled from a Gaussian and refines it over $K$ denoising steps, conditioned on the robot's current observations.

The noise-prediction network $\epsilon_\theta$ takes the current noisy action sequence $\mathbf{A}_t^k$, the observation $\mathbf{O}_t$, and the denoising iteration $k$ as input, and predicts the noise component. Subtracting this prediction yields a cleaner action sequence. Repeating for $K$ steps produces the final clean action trajectory $\mathbf{A}_t^0$.

The denoising step follows this update rule:

Training Diffusion Policy is elegant in its simplicity. From a dataset of demonstration trajectories, the process samples a clean action sequence $\mathbf{A}_t^0$, adds noise at a randomly chosen level $k$ to produce a corrupted version, and asks the network to predict the noise that was added. The loss function is a standard mean squared error:

Three design choices unlock the full potential of diffusion for robot control: the visual conditioning strategy, the receding-horizon execution scheme, and the choice between CNN and Transformer denoising backbones.

Visual conditioning. A naive approach would model the joint distribution of observations and actions $p(\mathbf{A}_t, \mathbf{O}_t)$, denoising both together. Diffusion Policy instead conditions on observations: it models $p(\mathbf{A}_t | \mathbf{O}_t)$. This means the visual encoder runs once per inference, not once per denoising step — a dramatic speedup that makes real-time control feasible. The visual features are injected into the denoising network through FiLM conditioning (for the CNN backbone) or cross-attention (for the Transformer backbone).

CNN vs Transformer backbone. The CNN backbone (a 1D temporal convolution network) works well out of the box for most tasks. Its inductive bias toward smooth signals, however, can over-smooth actions that need to change rapidly. The Transformer backbone handles sharp action changes better but requires more tuning. The paper recommends starting with the CNN and switching to the Transformer if performance is low on tasks with high-frequency action variation.

Visual encoder. A ResNet-18 with spatial softmax pooling (to preserve spatial information) and GroupNorm instead of BatchNorm (for compatibility with the exponential moving average used in DDPM training) encodes each camera view independently. The encoder is trained end-to-end with the diffusion policy.

One of the most impactful design choices is how the predicted action sequence is executed. The policy predicts $T_p$ future action steps but only executes the first $T_a$ of them before re-observing and re-planning. This is the receding-horizon control scheme, borrowed from model predictive control.

Think of it like driving: you look far down the road to plan your steering (prediction horizon $T_p$), but you only commit to the next few seconds of actual steering wheel movement (action horizon $T_a$) before checking the road again. This achieves two things: the long prediction horizon ensures temporal consistency and smooth trajectories, while the short execution horizon keeps the policy responsive to changes.

The observation horizon $T_o$ captures how many past frames the policy sees. Having $T_o = 2$ (the current and previous frame) often suffices, giving the policy implicit velocity information without computational overhead.

Diffusion Policy inherits several powerful properties from the diffusion framework, each addressing a known weakness in prior robot learning methods.

Multimodal action distributions. Multimodality arises naturally from two sources: the random initial noise $\mathbf{A}_t^K \sim \mathcal{N}(0, I)$ that determines which mode the trajectory converges to, and the stochastic noise added at each denoising step that allows exploration between modes. Unlike mixture-of-Gaussians policies that must specify the number of modes in advance, diffusion policies express arbitrary distributions — including ones with modes that appear and disappear depending on the observation.

Synergy with position control. Most behavior cloning methods use velocity control because position control amplifies multimodality problems (multiple valid positions exist at each step). Since Diffusion Policy handles multimodality well, it can exploit position control's advantage — less compounding error — and consistently outperforms the same architecture with velocity control.

Training stability. Energy-based policies (IBC) use InfoNCE loss with negative sampling to estimate the normalization constant. This estimation is noisy, causing training spikes and making checkpoint selection difficult. Diffusion Policy sidesteps this entirely by learning the score function (gradient of log-probability), which is independent of the normalization constant. The result: smooth training curves and reliable checkpoint selection.

Diffusion Policy was evaluated across 15 tasks from 4 benchmarks: RoboMimic (Lift, Can, Square, Transport, ToolHang), Push-T, Block Push, and Franka Kitchen. These tasks span single-arm and bimanual robots, 2-DoF to 14-DoF action spaces, rigid and fluid objects, and data from single or multiple human demonstrators.

The results were striking. Diffusion Policy outperformed prior state-of-the-art methods — LSTM-GMM (BC-RNN), IBC, and BET — on every task variant, with an average success-rate improvement of 46.9%. Particularly large gains appeared on tasks requiring precision (ToolHang: from 67% to 100%), multi-stage reasoning (Kitchen p4: from 44% to 99%), and multimodal behavior (Block Push p2: from 71% to 94%).

On real-world tasks — Push-T with a UR5 arm, sauce pouring and spreading with a Franka, mug flipping, and three bimanual tasks (egg beater, mat unrolling, shirt folding) — Diffusion Policy achieved robust performance with as few as 90-250 demonstrations, running at 10 Hz inference using DDIM with 10 denoising steps.

The paper provides a reassuring sanity check: in the simplest possible case — a linear dynamical system controlled by a linear feedback policy $\mathbf{a}_t = -\mathbf{K}\mathbf{s}_t$ — Diffusion Policy converges to the correct controller. The optimal denoiser in this case becomes $\epsilon_\theta(\mathbf{s}, \mathbf{a}, k) = \frac{1}{\sigma_k}[\mathbf{a} + \mathbf{K}\mathbf{s}]$, and DDIM sampling converges to $\mathbf{a} = -\mathbf{K}\mathbf{s}$. For trajectory prediction ($T_p > 1$), the denoiser implicitly learns the dynamics model to predict future actions as $\mathbf{a}_{t+t'} = -\mathbf{K}(\mathbf{A} - \mathbf{B}\mathbf{K})^{t'}\mathbf{s}_t$.

This means that to clone behavior that depends on state, the diffusion policy must implicitly learn a task-relevant dynamics model — an emergent property that helps explain the method's strong performance on contact-rich tasks where dynamics matter.

A critical requirement for robot control is speed. Full DDPM sampling with 100 denoising steps is too slow for closed-loop control. Diffusion Policy uses DDIM (Denoising Diffusion Implicit Models), which decouples the number of training and inference steps. Training with 100 noise levels but inferring with only 10 steps achieves 0.1 second inference latency on an Nvidia 3080 GPU — fast enough for 10 Hz real-time control.

The paper also uses a square cosine noise schedule (from iDDPM), which better captures both high-frequency and low-frequency characteristics of action signals compared to the original linear schedule.