Reflexion: Language Agents with Verbal Reinforcement Learning

By 2023, LLM-based agents like ReAct could interact with external environments — searching the web, navigating text-based games, writing and executing code. But when they failed a task, they had only two options, both bad.

The first option is traditional reinforcement learning: compute gradients and update the model's weights. For a model with billions of parameters, this is prohibitively expensive. You'd need large datasets, specialized hardware, and hours of compute — just to teach the agent one lesson.

The second option is to simply retry. But retrying without memory is like re-taking an exam without studying: you tend to make the same mistakes. The agent's behavior is determined entirely by its fixed weights and the current prompt. With no record of past failures, it has no reason to try a different approach.

Reflexion proposes a third path: let the agent talk to itself about what went wrong, save that self-talk, and read it before the next attempt.

Reflexion has three components that work together in an iterative loop:

The Actor ($M_a$) is the LLM agent that generates actions. It can be any action-generating strategy — Chain of Thought for reasoning, or ReAct for multi-step decision-making. The Actor receives the current task plus any stored reflections from memory, and produces a trajectory of actions and observations.

The Evaluator ($M_e$) scores the Actor's output. This could be as simple as a binary pass/fail from a compiler, an exact-match check against a ground-truth answer, or even an LLM-based judge. It produces a reward signal $r_t$ for trial $t$.

The Self-Reflection model ($M_{sr}$) is the heart of the framework. Given the failed trajectory $\tau_t$ and the reward $r_t$, it generates a verbal analysis: what went wrong, which specific action caused the failure, and what to do differently. This text is appended to the agent's episodic memory.

The critical design choice is how the policy is parameterized. In standard RL, the policy $\pi_\theta$ is the model weights $\theta$. In Reflexion, the policy is $\pi_\theta$ where $\theta = \{M_a, \text{mem}\}$ — the frozen LLM plus the episodic memory buffer. Learning happens by updating the memory, not the weights:

The self-reflection step is where Reflexion converts a sparse reward signal into rich, actionable feedback. Consider a coding agent that submits code which fails 3 out of 5 unit tests. A traditional RL agent gets the reward $r = 0.4$ and somehow has to figure out what to fix. The Reflexion self-reflection model instead produces text like:

"My implementation of the binary search used <= instead of < in the while condition, causing an infinite loop on edge cases. I also forgot to handle the empty array case. Next time, I should add boundary checks before the main loop and use < for the comparison."

This is vastly more informative than a scalar reward. It performs credit assignment — identifying which specific action caused the failure — and provides a concrete plan for the next attempt. The agent doesn't need to discover these insights through random exploration; the self-reflection model tells it directly.

Reflexion is designed to work with different types and sources of feedback signals. The framework is flexible — what matters is that there is some signal the agent can reflect on.

For decision-making tasks like AlfWorld, the evaluator uses hand-written heuristics. If the agent repeats the same action three times or exceeds 30 steps, it's likely stuck — time to reflect. The environment itself signals whether the task is complete.

For reasoning tasks like HotPotQA, the evaluator uses exact-match grading: does the agent's answer match the ground truth? A simple binary signal — but the self-reflection model turns it into rich verbal feedback about why the answer was wrong.

For programming tasks, the agent generates its own unit tests, then runs the code against them. The compiler output — error messages, stack traces, failed assertions — becomes the raw material for self-reflection. This is particularly powerful because the feedback is already detailed and specific.

Reflexion was tested on three fundamentally different types of tasks, each requiring different skills. Across all three, it substantially outperformed strong baselines — without any model fine-tuning.

Decision-making (AlfWorld): The agent navigates text-based household environments — find a spatula in a drawer, chill a tomato in the fridge. ReAct alone solved 75% of 134 tasks. ReAct + Reflexion solved 97% (130/134) across 12 iterative trials. The improvement came in two phases: a sharp initial spike (the agent quickly corrected obvious errors) and then a steady climb (it gradually learned to explore more systematically by using its memory to track which locations it had already checked).

Reasoning (HotPotQA): On 100 multi-hop questions requiring Wikipedia search and synthesis, Reflexion + ReAct improved accuracy by 20% over ReAct alone. Notably, the baseline agents showed zero improvement across retries (at temperature 0.7), while Reflexion agents steadily improved — proving that the self-reflection, not randomness, drove the gains.

Programming (HumanEval): Reflexion achieved 91% pass@1 accuracy on the HumanEval Python benchmark, surpassing GPT-4's 80%. The agent generates code, runs it against self-generated unit tests, reflects on failures, and iterates. This is where the framework shines brightest: compiler errors provide detailed, structured feedback that is ideal for self-reflection.

The authors conducted ablation studies to isolate the contribution of each component. The most revealing experiment compared three approaches on HotPotQA reasoning with ground-truth context:

1. Baseline retry — simply retry the task with no memory. Performance stays flat.

2. Episodic memory (EPM) — include the most recent trajectory as context, but no verbal reflection. This helps moderately — the agent can see what it tried before.

3. Reflexion — add the self-reflection step that distills the failure into a verbal lesson. This adds an 8% absolute improvement over EPM alone.

The gap between EPM and Reflexion demonstrates that raw experience replay is not enough. The distillation into verbal lessons is what makes the difference. The agent doesn't just see what it did — it understands why it failed and what to change.

For programming, removing self-generated tests cut accuracy from 91% to 82%. Without tests, the agent has no way to check its own work before submission. And removing self-reflection while keeping the tests dropped accuracy further — the tests alone provide raw error signals, but the self-reflection is needed to convert those signals into actionable plans.

Reflexion has clear limitations that the authors acknowledge. First, it relies on the LLM's ability to accurately identify its own mistakes — a form of self-evaluation that is not always reliable. If the model generates an incorrect self-reflection ("I failed because the API was slow" when the actual issue was a logic error), the bad lesson gets stored in memory and may hurt future attempts.

Second, the context window is a hard constraint. With memory capped at 1–3 reflections, the agent cannot accumulate unbounded experience. Older lessons are dropped, which means it may re-encounter and re-solve problems it had already learned from.

Third, there is no formal convergence guarantee. Traditional RL methods can prove convergence under certain conditions; Reflexion offers no such theoretical backing. Its effectiveness is empirical — it works well in practice, but we cannot prove it will always improve.

Despite these limitations, the core idea — that language can serve as a reinforcement signal — has proven remarkably durable and has influenced the design of nearly every subsequent agentic system.