High-Dimensional Continuous Control Using Generalized Advantage Estimation

In reinforcement learning, the agent collects trajectories — sequences of states, actions, and rewards — and uses them to improve its policy. The policy gradient tells us which direction to adjust the policy parameters so that future trajectories yield higher cumulative reward.

The catch: the raw policy gradient uses the total return following each action, and that return is the sum of many future rewards, each subject to the randomness of the environment and the policy. Like measuring the height of a wave in a storm, each measurement is valid but wildly variable. This high variance means you need millions of samples before the gradient signal rises above the noise.

A natural fix is to subtract a baseline — typically the value function $V(s)$ — from the return. What remains is the advantage function $A(s,a) = Q(s,a) - V(s)$: how much better (or worse) this specific action was compared to the average action from this state. The advantage is a much cleaner signal because the common baseline soaks up most of the randomness.

Before we can build GAE, we need its fundamental brick: the temporal-difference (TD) residual. After taking action $a_t$ in state $s_t$, the agent observes reward $r_t$ and lands in state $s_{t+1}$. The TD residual measures the surprise: how much better the actual one-step outcome was compared to what the value function predicted.

Think of it like checking your bank balance. You expected $V(s_t)$ in your account. After one transaction, you received $r_t$ and now your expected future balance is $V(s_{t+1})$. The residual $\delta_t$ is the discrepancy — the unexpected gain or loss.

If the value function $V$ were perfect, $\delta_t$ would be an unbiased estimate of the advantage. In practice $V$ is approximate (a neural network), so $\delta_t$ carries bias — but it's a single number per timestep, so its variance is low. This is the low-variance-high-bias extreme.

At the other extreme, you could ignore $V$ entirely and use the full Monte Carlo return — summing all actual rewards to the end of the episode. That's unbiased but has enormous variance because it depends on every future random event.

GAE gives us a smooth path between these two extremes.

The key insight is to take an exponentially-weighted average of multi-step advantage estimates. A one-step estimate uses one TD residual. A two-step estimate chains two residuals together. A k-step estimate chains k residuals. GAE blends all of them with weights that decay exponentially by the factor $\gamma\lambda$:

This is strikingly analogous to TD(λ) from temporal difference learning. In TD(λ), eligibility traces blend one-step and multi-step value updates. GAE applies the same exponential blending idea, but to the advantage function used in policy gradients rather than to value updates.

The elegance is in the simplicity: one parameter controls the entire bias-variance spectrum. In practice, $\lambda \approx 0.95$–$0.97$ tends to work well — mostly Monte Carlo–like (rich multi-step signal) but with just enough decay to tame the variance.

GAE has two parameters that both affect the bias-variance tradeoff, but they play different roles:

• γ (gamma) — the discount factor. It downweights rewards that are far in the future. Lowering γ makes the agent more short-sighted, reducing variance (fewer future rewards matter) but introducing bias (the agent ignores delayed consequences). Think of γ as the agent's time horizon for caring about outcomes.

• λ (lambda) — the trace decay. It downweights TD residuals that are far in the future. Lowering λ makes the advantage estimate rely more on the value function's predictions (low variance, high bias). Think of λ as how much you trust your value function versus raw experience.

In practice, both are set between 0.9 and 0.99. A common recipe: $\gamma = 0.99$, $\lambda = 0.95$.

The beauty of GAE is that its two boundary values recover two well-known estimators:

When λ = 0: the infinite sum collapses to just $\delta_t$, the one-step TD residual. This is the same as the simple advantage estimate $r_t + \gamma V(s_{t+1}) - V(s_t)$. It has low variance because it depends on a single transition, but high bias if the value function is imperfect.

When λ = 1: the sum becomes $\sum_{l=0}^{\infty} \gamma^l \delta_{t+l}$, which telescopes into $\sum_{l=0}^{\infty} \gamma^l r_{t+l} - V(s_t)$ — the discounted Monte Carlo return minus the baseline. Zero bias (given the correct discount), but high variance because it uses every future reward.

Any $\lambda$ between 0 and 1 gives you a point on the continuous spectrum between these extremes.

Having a good advantage estimator is only half the battle. The paper's second contribution is applying trust region optimization to both the policy network and the value function network.

For the policy, TRPO constrains each update so the new policy stays close to the old one (measured by KL divergence). This prevents catastrophic policy collapses where one bad update ruins everything.

For the value function, the paper uses a similar trust-region constraint. This is important because the value function appears inside the advantage estimate: if the value function changes too wildly between iterations, the advantage signal becomes unreliable. Think of it as calibrating your measuring instrument before every measurement — if the instrument drifts, your readings are meaningless.

A subtle point: the value function must be fitted after computing the policy gradient, using the old value function for the advantage. If you updated $V$ first and it overfitted, the TD residuals $\delta_t$ would all become near-zero, and the policy gradient would vanish.

The policy gradient tells us how to nudge the policy parameters to increase expected reward. With GAE plugged in as the advantage estimator, the gradient becomes:

Read this formula as a simple instruction to the optimizer: for each action the agent took, if the GAE advantage says it was better than average, increase its probability; if worse, decrease it. The magnitude of the advantage determines how aggressive the update is. Because GAE controls variance, these updates are stable enough to make steady progress on complex tasks.

The paper demonstrated GAE on some of the hardest continuous control tasks of its time, using the MuJoCo physics simulator:

• Bipedal walker — a simulated humanoid learning to walk and run. With 33 state dimensions and 10 action dimensions, the agent must coordinate hip, knee, and ankle joints simultaneously.

• Quadrupedal runner — a four-legged robot learning a galloping gait. Even more joints and contacts to coordinate.

• Standing up — a humanoid starting on the ground and learning to stand. This requires a multi-phase strategy that pure one-step methods struggle with.

The key finding: λ = 0 (one-step TD) performed poorly on all tasks due to excessive bias, while λ between 0.9 and 0.99 with appropriate γ produced the best learning curves. This validated that the intermediate bias-variance point is crucial for hard problems.

Putting all the pieces together, the GAE-based policy optimization algorithm repeats the following loop:

GAE's most visible legacy is Proximal Policy Optimization (PPO), published by the same lead author one year later. PPO replaced TRPO's complex second-order optimization with a simple clipped surrogate objective, but kept GAE as its advantage estimator unchanged. Today, virtually every PPO implementation — from game-playing agents to robotic manipulation to RLHF for language models — computes advantages using the exact formula from this paper.

The influence extends further:

• RLHF pipelines for Claude, GPT, and other language models use PPO with GAE to compute advantages when fine-tuning from human preference data.

• Robotics — real-world robot learning systems routinely use GAE when training locomotion and manipulation policies.

• Game AI — OpenAI Five (Dota 2) and other complex game agents relied on GAE for stable training across billions of environment steps.