RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

Before RT-2, robotic control faced a fundamental knowledge barrier. Models like RT-1 were trained on tens of thousands of robot demonstrations and could reliably pick up an apple or move a can to the left. But this knowledge was procedural only — the robot learned motor patterns, not concepts.

Ask RT-1 to "pick up the object you would use as a hammer" and it has no idea what to do. It never saw that instruction in training. It does not know that hammers are heavy and hard, that a rock shares those properties, or that an energy drink is what a tired person needs. All of this is world knowledge — the kind that vision-language models absorb from billions of web pages.

Meanwhile, large vision-language models like PaLI-X and PaLM-E could answer "which object here could serve as a hammer?" with ease — but they could not move a robot arm. They operated in the world of words, not the world of physics. The missing bridge: how do you pour that semantic knowledge into the motor commands themselves?

RT-2's breakthrough rests on a disarmingly simple idea: treat robot actions as text tokens. A robot arm action has 8 dimensions — positional displacement in x, y, z, rotational displacement in roll, pitch, yaw, gripper extension, and a termination flag. Each continuous dimension is divided into 256 uniform bins, turning it into an integer. The full action becomes a string like "1 128 91 241 5 101 127" — and as far as the language model is concerned, this is just another sentence to generate.

This means you do not need a new architecture. You take an existing vision-language model that already understands images and text, and you teach it one more "language": the language of robot movements. The model's input is a camera image of the robot's workspace plus a natural language instruction like "pick up the blue block." Its output is a sequence of action tokens that the robot executes.

The choice of which tokens represent actions depends on the backbone. PaLI-X already has unique tokens for integers 0–999, so actions simply reuse the number tokens. PaLM-E does not have convenient integer tokens, so the 256 least frequently used tokens in its vocabulary are overwritten with action meanings — a form of symbol tuning where existing symbols acquire new semantics through training.

The training prompt follows a simple Q&A format: "Q: what action should the robot take to [pick up the blue block]? A: 1 128 91 241 5 101 127". From the model's perspective, this is no different from visual question answering — the same loss function, the same autoregressive token generation.

RT-2 is not a single model — it is a recipe applied to two different vision-language model backbones. Think of it as a cooking technique that works with different base ingredients:

RT-2-PaLI-X uses PaLI-X as its backbone — a ViT-22B visual encoder paired with a 32B encoder-decoder text model, totaling up to 55 billion parameters. The visual encoder splits the camera image into patches (just like ViT), projects them into tokens, and the encoder-decoder generates the action string autoregressively.

RT-2-PaLM-E uses PaLM-E — a 12 billion parameter decoder-only model that weaves visual tokens directly into the text token stream. It processes images and text in a single unified sequence, producing actions as the continuation of that sequence.

Both models take the same input — a camera image plus an instruction — and produce the same output: a string of 8 action tokens. The key finding: the recipe works regardless of backbone architecture, and bigger models generalize better.

A naïve approach would be to simply fine-tune the VLM on robot data — replace web images and text with robot camera images and action strings. But this causes catastrophic forgetting: the model loses its web knowledge, the very thing we want to transfer.

RT-2's solution is co-fine-tuning: in every training batch, robot trajectory data and original web data (visual question answering, image captioning, etc.) are mixed together. The robot data is up-sampled to occupy 50–66% of each mini-batch, ensuring the model gets enough action examples to learn control, while the remaining web data acts as an anchor that prevents the model from forgetting what a hammer looks like or what "tired" means.

This is conceptually similar to how language model fine-tuning mixes instruction data with original pre-training data — but here the stakes are different. If the model forgets "red" it cannot follow "pick up the red cup." The co-training ratio is not just a training detail; it is the mechanism by which web knowledge flows into robotic control.

The most striking result of RT-2 is not improved performance on known tasks — it is the emergence of entirely new capabilities that were never present in the robot training data. These capabilities come purely from the VLM's web knowledge flowing into robotic control:

• Symbol understanding: "Move the apple to the number 3" — the robot was never trained on numbers, but the VLM knows what "3" looks like.
• Semantic reasoning: "Pick up something you could use as a hammer" — the robot grabs a rock, not a sponge, because the VLM knows that hammers are heavy and hard.
• Human recognition: "Give the snack to the person on the left" — the VLM can distinguish people in the scene.
• Relative reasoning: "Pick up the smallest object" or "the one closest to the red block" — spatial and comparative reasoning from web pretraining.
• Common-sense inference: "Which drink would help a tired person?" — the robot picks up an energy drink, applying world knowledge no one ever demonstrated to it.

Inspired by chain-of-thought prompting in language models, the authors trained a variant of RT-2 that generates a natural language reasoning plan before outputting action tokens. Instead of input → action, the flow becomes input → plan → action.

For example, given the instruction "pick up something to help clean up a spill," the model first outputs: "I would pick up the sponge because it can absorb liquid." Then it emits the action tokens to grab the sponge.

This is not merely explanatory — the planning step actually improves task success, because the model can decompose multi-step reasoning into explicit intermediate steps. And crucially, the reasoning is inspectable: an operator can read the robot's plan before it acts, catching errors before they become physical mistakes.

The evaluation covered more than 6,000 real-robot trials across three categories:

Seen tasks — instructions and objects from the training set. RT-2 matched RT-1's strong performance (both above 90% success), confirming that co-fine-tuning does not sacrifice known-task reliability.

Generalization — same task types but with novel objects, backgrounds, or environments not seen during training. RT-2 approximately doubled RT-1's success rate, showing that VLM knowledge transfers to manipulation even when the exact visual scenario is new.

Emergent capabilities — entirely new task categories requiring symbol understanding, semantic reasoning, or human recognition. RT-2 achieved over 3× the average success rate of the best baseline (RT-1 and VC-1). The 55B PaLI-X variant consistently outperformed the 12B PaLM-E variant, confirming that larger VLM backbones carry more transferable knowledge.

On the open-source Language Table benchmark, RT-2 reached 90% success versus the prior state-of-the-art of 77%.

RT-2's deepest contribution is not a benchmark number — it is the proof that the foundation model revolution applies to physical robots, not just text and images. The same recipe that made language models powerful — massive pre-training on web data, then task-specific fine-tuning — works for robotic control. And the connection is literal: actions are tokens, and knowledge flows through the same weights that process language.