Do As I Can, Not As I Say: Grounding Language in Robotic Affordances

Large language models like PaLM and GPT-3 encode an impressive wealth of procedural knowledge. Ask one how to clean a spill and it produces a sensible multi-step narrative. But that narrative was learned from text, not from physical interaction. The model has no concept of what the robot's gripper can reach, which objects are currently visible, or whether the suggested tool even exists in the scene.

On the other side, robots equipped with learned visuomotor policies can reliably execute short skills — "pick up the sponge", "go to the table" — but they cannot decompose a vague instruction like "I just worked out, can you bring me a drink and a snack to recover?" into an ordered sequence of those skills. They lack the world knowledge and commonsense reasoning that language models provide.

The fundamental tension is between two complementary weaknesses: the language model knows what to do but not what is possible; the robot knows what is possible but not what to do. SayCan resolves this by treating the problem as a product of two probabilities.

SayCan stands for two complementary functions working together. The Say component uses a large language model as a scoring function. Given a high-level instruction like "I spilled my drink, can you help?", the LLM is presented with each available skill description (e.g. "find a sponge", "pick up the coke can", "go to the trash") and outputs the probability that each skill is a useful next step toward completing the instruction. This is the task-grounding: it connects abstract language to concrete skill names.

The Can component uses a learned value function — an affordance function — that captures the probability of successfully executing a skill in the current physical state. For instance, "pick up the sponge" has a high affordance value when the sponge is visible and reachable, but near zero when the robot is navigating an empty hallway. This is the world-grounding: it connects skill names to physical reality.

The combined probability of each skill making real progress on the instruction is the product of these two terms. At each step, the robot selects the skill with the highest combined score, executes it, observes the new state, and repeats until the LLM outputs a termination token. The result is an interpretable, step-by-step plan expressed entirely in natural language.

Instead of asking the LLM to generate free-form text and hoping it names valid skills, SayCan uses the LLM as a scoring model. The key shift: rather than sampling open-ended completions, the system feeds each skill description as a candidate completion and reads the probability the LLM assigns to it. This constrains outputs to the set of skills the robot actually has.

The interaction is structured as a dialog. The user provides a high-level instruction ("How would you clean up after my lunch?") and the LLM produces a numbered step plan ("1. find a sponge, 2. pick up the sponge, 3. go to the table, 4. ..."). Prompt engineering provides a few examples of this dialog structure so the LLM knows the format. At each step, SayCan scores all available skills, picks the top one, appends it to the running plan, and queries the LLM again for the next step.

This scoring approach has a crucial advantage over generation: it is interpretable. The user can inspect the LLM's probability for every candidate skill and understand why the robot chose its plan.

The affordance function answers a simple question: "If I ask the robot to do this skill right now, will it succeed?" This is exactly what a value function provides in the reinforcement learning framework. The robot receives a reward of 1.0 if it successfully completes a skill and 0.0 otherwise, making the value function a direct estimate of the probability of completion.

The authors trained language-conditioned value functions via temporal-difference reinforcement learning. A single multi-task model accepts both the current camera image and a language embedding of the skill description, then outputs a Q-value between 0 and 1. When the sponge is right in front of the robot, "pick up the sponge" gets a high Q-value. When the robot is in an empty hallway, every "pick up" action gets a near-zero Q-value.

Think of the value function as the robot's spatial awareness: it forms a map of what is possible here and now across all 551 skills. This map changes with every step the robot takes, providing continuous feedback without needing explicit object detectors or scene descriptions. The SayCan system used both behavioral cloning policies (for better execution success) and RL-trained value functions (for better affordance estimation), combining the strengths of both approaches.

SayCan builds plans iteratively, not all at once. At each decision point, it scores all skills, selects the best, executes it, and then asks the LLM again — this time including the skill just executed in the conversation history. This iterative approach has two advantages. First, the LLM can reason about what has already been done: after delivering a drink, it knows to move on to the snack rather than fetching another drink. Second, the affordance values update after each action because the robot's physical state has changed — it may have moved to a new location or picked up an object.

Consider the instruction: "I just worked out, can you bring me a drink and a snack to recover?" SayCan must infer that "recover from a workout" means something healthy — water and an apple, not soda and chips. It must also sequence the subtasks: navigate to the water, pick it up, bring it, put it down, then navigate to the apple, pick it up, bring it, put it down, and terminate. At each step the LLM provides commonsense reasoning about what to do next, and the value functions ensure the chosen skill is currently feasible.

SayCan's skill repertoire spans seven families of robotic behaviors: picking up objects, placing objects, going to locations, finding objects, putting objects down, knocking objects over, and placing objects in specific configurations. Across 17 common kitchen objects and 5 locations (two counters, a table, a trash can, and the user location), this produces 551 distinct skills.

Skills were trained using two complementary methods. Behavioral cloning (BC) used 68,000 human teleoperated demonstrations collected over 11 months with a fleet of 10 robots. Reinforcement learning (RL) used the MT-Opt framework in simulation with RetinaGAN sim-to-real transfer. The BC policies achieved higher execution success rates, while the RL-trained value functions provided better affordance estimation — so SayCan used BC for execution and RL for scoring.

Both policy types are conditioned on language through a pre-trained Universal Sentence Encoder. The language embedding specifies which skill to perform, and a single multi-task network handles all skills within a family. This means adding a new skill requires only collecting demonstrations and adding a language label — no architecture changes.

SayCan was evaluated on 101 real-world instructions spanning 7 families: single primitives, abstract nouns, abstract verbs, structured language, embodiment-dependent tasks, crowd-sourced queries, and long-horizon plans. Using PaLM (540B parameters) as the LLM, PaLM-SayCan achieved 84% planning success and 74% execution success in a mock kitchen, and 81% planning / 60% execution in a real office kitchen.

The ablation studies reveal the critical importance of both components. Without value function grounding (using only the LLM's top skill), planning success dropped to 67%. Without the LLM (using only behavioral cloning with the raw instruction), success was 0% on all multi-step tasks. This confirms the core thesis: neither component alone is sufficient, but together they produce a capable robotic system.

Particularly notable is the LLM scaling result: switching from the 137B FLAN model to the 540B PaLM model improved planning from 70% to 84% and execution from 61% to 74%. This was the first demonstration that improving a language model directly improves robot performance — an exciting bridge between NLP research and robotics.

SayCan demonstrated several extensions beyond basic planning. Adding new skills — such as drawer manipulation (open, close, put in, take out) — required only adding the skill descriptions as LLM options and providing value functions. With just a single new prompt example, PaLM-SayCan achieved 100% planning accuracy on drawer tasks.

A known weakness of vanilla SayCan was handling negation ("bring me a snack that is not an apple") and reasoning-heavy queries. By integrating chain-of-thought prompting, where the LLM first generates an explicit explanation before scoring skills, these problems were substantially improved. For example: "Can you bring a fruit-flavored drink without caffeine?" triggers the explanation "The user wants a drink that is fruit-flavored and has no caffeine, I will bring the lime soda" — correctly resolving the constraint.

Finally, since PaLM was trained on multilingual data, SayCan handled queries in Chinese, French, and Spanish with almost no performance drop — despite never being explicitly designed for multilingual operation.

SayCan inherits the limitations of both its components. From the LLM side, it struggles with negation, ambiguous references (e.g. "something with caffeine"), and occasionally terminates long plans prematurely — bringing one object but forgetting the second. Of all planning errors, 65% were traced to LLM failures and 35% to affordance misclassification.

From the skill side, the system is fundamentally limited by the repertoire of available skills. It cannot improvise actions outside its training distribution. If no skill matches what the instruction requires, no amount of LLM reasoning will help.

The system also lacks closed-loop replanning: if a skill fails mid-execution, SayCan has no mechanism to detect the failure and retry or replan. The Inner Monologue paper later addressed this limitation by incorporating environment feedback into the LLM's planning loop.

SayCan's most enduring contribution may not be the specific algorithm, but the conceptual bridge it built. It showed that the vast knowledge in language models is not locked behind a text interface — it can be extracted, grounded, and made physically actionable. The idea that improving NLP directly improves robotics opened a research trajectory that continues to accelerate, from RT-1's data-driven approach to RT-2's unified vision-language-action model and beyond.