RT-1: Robotics Transformer for Real-World Control at Scale

In NLP and computer vision, the recipe for success became clear: collect a massive, diverse dataset, train a high-capacity model once, and then deploy or fine-tune it for many tasks. GPT absorbs text from the entire internet; CLIP learns from 400 million image-text pairs. These models generalize because they've seen enough variety.

Robotics had not yet made this leap. Each task — pick up a sponge, open a drawer, push a bottle — required its own dataset of demonstrations and its own trained policy. Gato (DeepMind, 2022) attempted a generalist agent, but its real-world robotics capability was limited to a single block-stacking task. Instruction-following methods existed, but their generalization to new objects or environments was weak.

The two key bottlenecks were data and architecture. Robot data is expensive — it requires physical hardware, human operators, and real-world time. And robot policies must run in real time on the robot itself, which limits model size. RT-1 attacks both problems simultaneously.

RT-1's architecture is a pipeline of three stages, each solving a different part of the problem. Think of it as an information highway where the camera image enters as a wide river of pixels, the natural language instruction acts as a traffic sign telling the river where to go, and the output is a compact set of motor commands.

Stage 1 — Visual-Language Fusion (FiLM-EfficientNet): The robot's camera image (300×300 pixels) enters an EfficientNet-B3 backbone, pre-trained on ImageNet. But how does the network know which task to perform? FiLM (Feature-wise Linear Modulation) conditioning injects the language instruction — encoded by a pre-trained Universal Sentence Encoder — directly into the convolutional layers. FiLM modulates each feature map with a task-specific scale and shift, essentially telling the visual network "pay attention to the green chip bag, not the red can."

Stage 2 — Token Compression (TokenLearner): After FiLM-EfficientNet, each image produces a 9×9 spatial grid of 512-dimensional features — 81 tokens total. For a 6-frame image history, that's 486 tokens — far too many for real-time Transformer inference. TokenLearner learns to compress these 81 tokens down to just 8 per image by learning spatial attention masks that select the most informative regions. This reduces the Transformer's input from 486 tokens to just 48, making 3 Hz inference feasible on the robot.

Stage 3 — Temporal Reasoning (Transformer): The 48 tokens enter a decoder-only Transformer with 8 self-attention layers. The Transformer attends across the 6-frame history, learning temporal patterns — like "the arm is moving toward the object" or "the gripper just closed." Its output is a sequence of action tokens.

A robot arm moves in continuous space — joint angles are real numbers, gripper openings are fractions. But Transformers excel at discrete sequences: picking one token from a vocabulary, like choosing a word from a dictionary. RT-1 bridges this gap through action tokenization.

Each action has 11 dimensions: 7 for the arm (x, y, z position, roll, pitch, yaw rotation, and gripper opening), 3 for the mobile base (x, y translation and yaw rotation), and 1 mode-switching command (to terminate the episode). Each continuous dimension is discretized into 256 bins, uniformly spanning the range of possible values. So "move the arm 3 cm to the right" becomes token 147 in the x-dimension bin.

This tokenization converts the control problem into a classification problem: at each timestep, the Transformer predicts 11 categorical distributions (one per dimension), each over 256 classes. This is trained with standard cross-entropy loss — the same objective used to train language models.

The dataset is arguably RT-1's most important contribution. Over 17 months, a fleet of 13 Everyday Robots mobile manipulators collected roughly 130,000 demonstration episodes across more than 700 task instructions. These tasks fall into categories such as picking, placing, opening and closing drawers, rotating objects, and pulling napkins. The instructions are natural-language commands like "pick rxbar chocolate from middle drawer" or "place green jalapeno chip bag to paper bowl."

The paper demonstrates that both data quantity and data diversity matter for generalization. Training on 10× more data improves performance, but adding diverse task types matters even more. A model trained on many simple picking tasks generalizes to novel objects better than one trained on fewer but more complex tasks. The key insight is that structural overlap between tasks — grasping is common to picking, placing, and drawer-opening — enables the model to compose learned skills in new ways.

RT-1 was evaluated on over 3,000 real-world robot trials — not in simulation. The results demonstrate three forms of generalization. First, new task generalization: RT-1 achieves 76% success on completely unseen task instructions, compared to 53% for Gato and 24% for BC-Z (the next-best imitation learning baseline). Second, robustness to distractors: when novel objects are placed on the table as distractions, RT-1 maintains 83% success vs. 47% for Gato. Third, background generalization: when evaluated in unseen kitchens with different lighting and layouts, RT-1 reaches 59% success vs. 41% for the nearest baseline.

The paper also shows that RT-1 integrates seamlessly with SayCan, a framework where a large language model plans a sequence of high-level steps (e.g., "I spilled my drink" → find sponge → pick up sponge → bring to table → wipe), and RT-1 executes each step. With RT-1 as the low-level controller, SayCan completes long-horizon tasks with up to 50 sequential stages — a dramatic increase over prior work.

Can RT-1 benefit from data that doesn't come from its own robot? The paper tests two scenarios. First, mixing in simulated episodes from a different visual domain. Second, adding real demonstrations from a Kuka IIWA arm — a completely different robot morphology. In both cases, the additional data does not hurt performance on the original tasks and improves generalization to new scenarios. This is a crucial result: it means the path to better robot intelligence is not just collecting more data on one robot, but pooling experience across robots, simulators, and even embodiments — a vision that culminated in the Open X-Embodiment project.

The paper ablates several architectural decisions. TokenLearner is essential — without it, the model is too slow for real-time control or must be made so small it can't learn the task diversity. With 8 tokens per image (down from 81), inference runs at 3 Hz while maintaining or improving performance. Image history of 6 frames outperforms single-frame input, as temporal context helps resolve ambiguities (is the arm approaching or retreating?). Action discretization into 256 bins outperforms both continuous regression and coarser discretizations. And FiLM conditioning is superior to late fusion of language and vision.

RT-1's lasting contribution is not any single architectural trick — it is the demonstration that the foundation-model recipe works for real-world robotics. Large diverse data + high-capacity model + simple training objective = a robot that generalizes. This insight now drives the entire field: from RT-2's web-knowledge transfer to Open X-Embodiment's cross-institution data pooling to the latest vision-language-action models. Every time a research lab trains a general-purpose robot policy on diverse demonstrations, the lineage traces back to RT-1.