RAFT: Recurrent All-Pairs Field Transforms for Optical Flow

Optical flow is the task of estimating a dense displacement field between two consecutive frames of a video. For every pixel $(u, v)$ in frame $I_1$, the goal is to find a displacement vector $(f^1, f^2)$ such that the pixel's corresponding location in frame $I_2$ is $(u + f^1, v + f^2)$. The result is a flow field — a map of arrows that describes how every pixel moved.

This is not about tracking a few special points; it is about computing motion for every single pixel in the image. A 1080p frame has over two million pixels, each needing its own displacement vector. Think of it as painting an arrow on every dot in a pointillist painting, showing which way that dot drifted between two moments in time.

Optical flow is used in video stabilization, action recognition, autonomous driving, video editing, and medical imaging. It is a foundational building block for any system that needs to understand motion in video.

RAFT's design can be distilled into three stages, each clean and self-contained:

Stage 1 — Feature Extraction. A shared convolutional network encodes both input images into compact per-pixel feature vectors at 1/8 resolution. A second context encoder processes only the first image and produces features that guide the update operator.

Stage 2 — Correlation Volume. The dot product between every pair of feature vectors from the two images produces a 4D correlation volume of size $H \times W \times H \times W$ (at 1/8 resolution). This volume is then average-pooled on the last two dimensions to form a four-level correlation pyramid that captures similarity at multiple spatial scales.

Stage 3 — Iterative Updates. A convolutional GRU starts from an initial flow of zero and, at each iteration, looks up a local patch from the correlation pyramid around its current flow estimate, ingests correlation features together with context features, and produces a flow update $\Delta f$. After many iterations, the flow converges to an accurate estimate.

All three stages are differentiable and trained end-to-end.

RAFT uses two encoder networks to convert raw images into compact representations:

Feature Encoder ($g_\theta$) — Applied to both images $I_1$ and $I_2$. It is a convolutional network with residual connections that maps each $H \times W \times 3$ image to a $\frac{H}{8} \times \frac{W}{8} \times D$ feature map (where $D = 256$). The encoder uses instance normalization — this is important because instance normalization operates per-image, making features invariant to global brightness or contrast shifts between the two frames. The same weights are shared for both images. Think of this as giving both photographs to the same analyst who applies the same interpretation rules to each one.

Context Encoder ($h_\theta$) — Applied only to $I_1$. It has the same architecture but uses batch normalization instead of instance normalization, producing context features and an initial hidden state for the GRU. The context encoder captures information specific to the reference frame — edges, texture boundaries, semantic cues — that helps the update operator decide how to refine the flow at each location. If the feature encoder is like a translator converting images to a common language, the context encoder is like a local guide who knows the terrain of the first image.

The correlation volume is the heart of RAFT. Given feature maps $g_\theta(I_1)$ and $g_\theta(I_2)$, the correlation volume is computed by taking the dot product of every feature vector in the first image with every feature vector in the second image. If both feature maps have spatial dimensions $H' \times W'$ (where $H' = H/8$, $W' = W/8$), the result is a 4D tensor:

Think of the correlation volume as a massive phone book. For each pixel in the first image, you can look up how similar it is to any pixel in the second image. High values mean strong matches; low values mean poor matches. Instead of searching the entire phone book every time, RAFT builds a multi-scale index — the correlation pyramid.

A single correlation volume captures fine-grained similarity but has a limited receptive field — it only directly shows matches within a local neighborhood. To handle large displacements (fast-moving objects), RAFT constructs a correlation pyramid by repeatedly average-pooling the last two dimensions of $C$ with kernel sizes 1, 2, 4, and 8:

At level 0, the volume has full $H' \times W'$ resolution on the second image's dimensions. At level 3, those dimensions are reduced to $\frac{H'}{8} \times \frac{W'}{8}$. Crucially, the first two dimensions (the $I_1$ dimensions) are never pooled — this preserves per-pixel resolution in the reference frame.

The key insight: lower pyramid levels have a wider effective receptive field, capturing large displacements. Higher levels preserve fine spatial detail for small, precise motions. Together, they cover both scenarios. This is fundamentally different from traditional coarse-to-fine approaches because the pyramid is built from a single correlation volume at one feature resolution, not from separate feature extractors at multiple resolutions.

Correlation Lookup. At each iteration, the update operator needs to know "what does the correlation volume say about my current flow estimate?" For each pixel $\mathbf{x}$ in $I_1$, the current flow maps it to an estimated correspondence $\mathbf{x}' = \mathbf{x} + \mathbf{f}$. The lookup operator places a grid of radius $r$ around $\mathbf{x}'$ in the correlation volume and bilinearly samples from it. This is done at every level of the pyramid, and the results are concatenated into a single feature vector.

Think of it as using your current guess to check a specific neighborhood in the phone book: "given where I think this pixel went, what do the nearby entries say about that guess?" The multi-level lookup means you check at multiple zoom levels simultaneously — a coarse overview and a fine-grained local view — giving the update operator both global context and local precision.

The update operator is the engine of RAFT. It is inspired by classical optimization algorithms that iteratively refine a solution. Starting from an initial flow $\mathbf{f}_0 = \mathbf{0}$, it produces a sequence of improved estimates $\{\mathbf{f}_1, \mathbf{f}_2, \ldots, \mathbf{f}_N\}$ where each update is:

At each iteration $k$, the update operator performs four steps:

1. Correlation lookup — uses $\mathbf{f}_k$ to sample a local patch from all pyramid levels.
2. Input assembly — concatenates the correlation features, the current flow (encoded through two convolution layers), and the context features from the context encoder.
3. GRU update — passes the assembled input through a convolutional GRU that updates a hidden state $\mathbf{h}_k$. The GRU has separate gates for horizontal and vertical spatial patterns (using $1 \times 5$ and $5 \times 1$ convolutions).
4. Flow prediction — passes $\mathbf{h}_k$ through two convolutional layers to produce $\Delta \mathbf{f}_k$.

The GRU's hidden state carries memory across iterations — it "remembers" what it has learned in previous refinement steps. This is like a detective who does not start from scratch each round but builds on accumulated evidence. The GRU weights are shared across all iterations, so the total parameter count stays small (5.3M for the full model).

The update operator produces a flow field at 1/8 resolution. To recover full-resolution flow, RAFT uses learned convex upsampling — a technique more principled than bilinear interpolation.

For each pixel in the full-resolution output, the network predicts a set of 9 weights (a $3 \times 3$ mask) that form a convex combination of the 9 nearest neighbors in the coarse flow field. The weights are predicted from the final hidden state of the GRU and passed through a softmax to ensure they sum to 1. This means each high-resolution pixel is a weighted average of its coarse neighbors, with the weights learned to respect motion boundaries.

Think of it as filling in a mosaic. Bilinear interpolation would blindly average the four nearest tiles. Convex upsampling asks the network: "these 9 coarse tiles are your neighbors — how should I blend them?" The network learns to give nearly all the weight to tiles on the same side of a motion boundary and almost zero weight to tiles on the other side, preserving sharp edges in the flow.

RAFT is trained with a loss function that supervises every iteration of the update operator, not just the final output. For a sequence of $N$ predicted flows $\{\mathbf{f}_1, \ldots, \mathbf{f}_N\}$ and ground-truth flow $\mathbf{f}_{gt}$:

Before RAFT, the dominant approach to optical flow was the coarse-to-fine pyramid. The idea was intuitive: estimate flow at low resolution first (where large motions become small), then warp the second image using that estimate, and refine at higher resolution. Repeat until you reach full resolution.

This approach has three critical weaknesses that RAFT avoids:

Error cascading. Mistakes at coarse levels propagate to finer levels and cannot be corrected. A wrong large-motion estimate at the top level will misalign the warped image for all subsequent levels.

Small object loss. Aggressive downsampling at coarse levels can shrink small objects below a single pixel, making their motion unrecoverable.

Architectural complexity. Each pyramid level needs its own feature extractor, cost volume, and estimator — multiplying parameters and training difficulty.

RAFT sidesteps all three by operating at a single resolution (1/8). The correlation pyramid is not built from multiple feature resolutions — it is built by pooling a single correlation volume. Large motions are captured by the coarse levels of the pyramid, and the recurrent updates can correct early mistakes because each iteration re-examines the correlation evidence from scratch.

RAFT set new records across multiple optical flow benchmarks:

On Sintel (final pass), RAFT achieved an end-point error (EPE) of 2.855 pixels — a 30% reduction from the previous best (4.098 pixels). On KITTI 2015, it achieved an F1-all error of 5.10% — a 16% reduction from the previous best (6.10%).

Beyond raw accuracy, three aspects stood out:

Cross-dataset generalization. When trained only on synthetic data (FlyingChairs + FlyingThings3D) and evaluated on KITTI without fine-tuning, RAFT achieved 5.04 EPE — a 40% improvement over the best prior method. This suggests the architecture learns genuinely transferable motion representations.

Training efficiency. RAFT converges in approximately 100K iterations — roughly 10× fewer than competing methods like FlowNet2 or PWC-Net.

Compact model. The full model has only 5.3M parameters. A smaller variant (RAFT-S) with 1.0M parameters still outperforms all prior methods on Sintel, running at 20 FPS on a 1080Ti GPU.

RAFT proved that you do not need complex multi-scale architectures to estimate dense correspondences. A single correlation volume, a lightweight recurrent updater, and iterative refinement — these three simple ideas, composed well, outperformed years of architectural engineering. The lesson extends beyond optical flow: when in doubt, prefer iterative refinement over single-shot prediction. Let the network correct its own mistakes.