Voyager: An Open-Ended Embodied Agent with Large Language Models

Minecraft is unlike most environments studied in AI. It has no predefined end goal, no fixed storyline — just an infinite procedurally generated world with endless possibilities. An effective agent in such a world needs three capabilities that existing approaches lack:

Propose suitable tasks based on its current skill level and world state. If it spawns in a desert, it should learn to harvest sand and cactus before attempting iron tools — not follow a rigid curriculum designed for a forest.

Refine and remember skills. When a program to fight zombies fails, the agent should debug it using environment feedback. Once mastered, the skill should be stored permanently and reusable for similar tasks (fighting spiders uses similar logic).

Explore continuously in a self-driven manner, seeking out new challenges without waiting for human instructions.

Classical reinforcement learning agents struggle with all three. They operate on low-level actions (move forward, turn left), making it hard to build temporally extended behaviors. Recent LLM-based agents like ReAct and Reflexion can reason and reflect, but they lack persistent memory — each episode starts from scratch.

The automatic curriculum is Voyager's internal teacher. At each step, GPT-4 receives the agent's current state — its inventory, position, biome, health, time of day, completed tasks, and failed tasks — and generates the next task to attempt. The overarching directive is simple: "discover as many diverse things as possible."

This is not a hand-designed curriculum. GPT-4 uses chain-of-thought prompting to reason about what the agent should try next, given what it already knows and what surrounds it. If the agent is near a river and has never fished, the curriculum might propose "catch a fish." If it has wood tools but no stone tools, it might propose "mine cobblestone." The curriculum adapts to the agent's actual trajectory rather than following a fixed script.

Additional context comes from a GPT-3.5-based question-answering module backed by the Minecraft Wiki. When GPT-4 generates a task, GPT-3.5 retrieves relevant crafting recipes, mob behavior, or biome information, grounding the curriculum in factual game knowledge. This two-tier LLM design separates high-level reasoning (GPT-4) from factual retrieval (GPT-3.5), balancing cost and capability.

The skill library is Voyager's long-term memory. Every time the agent successfully completes a task, the corresponding program is stored as a reusable skill. Each skill consists of executable JavaScript code (using the Mineflayer API) and a natural language description generated by GPT-4.

Storage: Skills are indexed by the embedding of their descriptions. When GPT-4 writes a new program, it also generates a one-line description like "mine 3 cobblestone blocks" or "craft a wooden pickaxe." This description is embedded using a text embedding model, and the resulting vector is stored alongside the code in a vector database.

Retrieval: When the curriculum proposes a new task, Voyager queries the skill library with the new task description, retrieving the top-5 most similar skills. These retrieved programs are injected into the prompt as examples, so GPT-4 doesn't start from scratch — it can reuse, adapt, or compose existing skills.

Compositionality: This is where the skill library becomes powerful. A skill for "craft a wooden pickaxe" can call the skill for "get 3 planks," which in turn calls "chop a tree." Complex behaviors emerge from composing simpler ones — like building blocks. This compositional structure means Voyager's capabilities grow super-linearly: each new skill can leverage all previous skills.

No catastrophic forgetting: Because skills are stored as code in an external database rather than encoded in neural network weights, Voyager never forgets. A skill learned in the first hour is just as accessible in the hundredth hour. This is a fundamental advantage over agents that store knowledge in model parameters.

When GPT-4 generates code for a new task, the program rarely works on the first try. Voyager uses an iterative refinement loop with three feedback channels:

Environment feedback: After executing the code, the game environment returns the agent's new state — updated inventory, position, health, nearby blocks and entities. If the task was "mine 3 iron ore" and the inventory shows only 1 iron ore, GPT-4 sees the gap and revises the code.

Execution errors: If the code throws a runtime error (e.g., trying to craft an "acacia axe" that doesn't exist in Minecraft), the error message is fed back to GPT-4, which debugs the code and produces a corrected version.

Self-verification: This is a separate GPT-4 call that acts as a critic. Given the task description and the agent's current state after execution, the verifier decides whether the task was truly completed. If not, it provides a critique explaining what went wrong and how to fix it. Only after self-verification passes is the skill committed to the library.

This three-channel feedback loop runs for up to 4 iterations per task. If the task still fails after all retries, it's added to a "failed tasks" list that the curriculum uses to avoid proposing tasks that are currently too difficult.

A critical design choice in Voyager is representing actions as executable code rather than low-level motor commands. In reinforcement learning, an agent typically acts through primitive operations — move forward, turn, click. Building complex behaviors from these primitives requires hundreds of sequential decisions, and the agent must re-learn the entire sequence for every new task.

Voyager writes JavaScript programs that call the Mineflayer bot API. A single "skill" program can contain loops, conditionals, function calls, and error handling — all expressed in a few lines of code. For example, "mine 3 iron ore" is a complete program that navigates to iron ore, mines it, checks the inventory, and repeats until 3 are collected.

This has three major advantages. Programs are temporally extended: one skill can represent minutes of gameplay. Programs are interpretable: a human can read the code and understand exactly what the agent is doing. And programs are compositional: complex skills naturally call simpler skills as subroutines.

Voyager was evaluated against three baselines — ReAct, Reflexion, and AutoGPT — across four dimensions in Minecraft. The results demonstrate the compounding power of lifelong learning with a skill library:

Exploration: Voyager discovered 63 unique items in 160 prompting iterations, 3.3× more than the best baseline. While baselines plateau quickly (repeating known behaviors), Voyager keeps discovering because its curriculum pushes toward novelty and its skill library enables increasingly complex actions.

Tech tree mastery: The Minecraft tech tree requires crafting tools in a hierarchy — wooden → stone → iron → diamond. Voyager unlocked the wooden level 15.3× faster, the stone level 8.5× faster, the iron level 6.4× faster, and was the only agent to reach the diamond level. Baselines could not compose the multi-step skills needed.

Map coverage: Voyager traversed 2.3× longer distances than baselines, exploring diverse biomes (deserts, forests, mountains) while baselines remained confined to local areas.

Zero-shot generalization: When placed in a completely new world with an empty inventory, Voyager used its skill library to solve novel tasks (e.g., "obtain a diamond pickaxe") that it had never seen during training. Baselines could not solve any of these tasks within 50 iterations. Remarkably, even giving the Voyager skill library to AutoGPT boosted its performance, showing the library works as a plug-and-play asset.

The ablation studies reveal that each of Voyager's three components contributes significantly to its performance:

Removing the automatic curriculum and replacing it with a fixed task list causes a sharp drop — the agent can't adapt to its environment and wastes iterations on tasks that don't match its current state.

Removing the skill library means the agent starts from scratch every time. Without the ability to retrieve and compose previously mastered programs, complex tasks become unreachable.

Removing self-verification leads to a library polluted with buggy, incomplete programs. The agent adds skills that don't actually work, and subsequent tasks that compose those skills fail in cascade.

Additionally, replacing GPT-4 with GPT-3.5 for code generation significantly degrades performance, showing that the quality of the underlying language model is crucial for generating correct, complex programs.

Voyager's deepest contribution is architectural, not domain-specific. The three-component pattern — a curriculum that proposes tasks, a library that stores solutions, and a verifier that ensures quality — is a general blueprint for lifelong learning agents. The same architecture could power a coding assistant that builds up a library of reusable code patterns, a robotic system that accumulates manipulation skills, or a research agent that composes investigative strategies.

The key insight is separating knowledge storage from model parameters. By externalizing skills as code in a retrievable database, Voyager achieves something neural networks alone cannot: it never forgets, it always compounds, and its knowledge is transferable across environments.