Proximal Policy Optimization Algorithms

In reinforcement learning, a policy $\pi_\theta(a|s)$ maps states to action probabilities. The agent collects trajectories, computes returns, and updates $\theta$ to increase the probability of good actions. This is the policy gradient idea behind REINFORCE.

The core difficulty: how far should each update go?

• Too small and learning is painfully slow — every trajectory is used once then discarded.
• Too large and the policy can collapse: it overshoots into a bad region, collects bad data from there, and spirals downward. One bad update can undo thousands of good ones.

This is not a theoretical fear. In practice, vanilla policy gradient training is famously unstable, and tuning the learning rate is a dark art.

Trust Region Policy Optimization solved the overshoot problem by adding a hard constraint: the KL divergence between the old and new policies must stay below a threshold $\delta$. This guarantees monotonic improvement in theory — every update either improves the policy or does nothing.

But TRPO pays a price for this guarantee:

• It requires computing the Fisher information matrix (second-order derivatives), making each step expensive.
• The constrained optimization uses conjugate gradient with a line search — complex to implement and debug.
• It is incompatible with architectures that share parameters between policy and value networks, and with regularization techniques like dropout.

PPO asks: can we get TRPO's stability with nothing more than first-order gradients and a clever objective function?

PPO starts from the same surrogate objective that TRPO uses, but replaces the hard constraint with a remarkably simple mechanism. First, define the probability ratio:

$r_t(\theta) = \frac{\pi_\theta(a_t | s_t)}{\pi_{\theta_\text{old}}(a_t | s_t)}$

This ratio measures how much the new policy differs from the old one for a specific action. When $r = 1$, the policies agree perfectly. When $r > 1$, the new policy assigns more probability to this action; when $r < 1$, less.

The unconstrained surrogate objective would be:

$L^{CPI}(\theta) = \hat{\mathbb{E}}_t \left[ r_t(\theta) \, \hat{A}_t \right]$

where $\hat{A}_t$ is the estimated advantage — how much better this action was compared to the average. Maximizing this objective pushes the policy to take good actions more often. But without a constraint, nothing stops $r_t(\theta)$ from growing unboundedly, leading to destructively large updates.

Why the minimum? Think of it as a pessimistic bound. For actions with positive advantage ($\hat{A}_t > 0$ — actions better than average), the clipped version caps the benefit when $r$ exceeds $1+\varepsilon$: "yes, this action is good, but you've already increased its probability enough." For actions with negative advantage ($\hat{A}_t < 0$ — actions worse than average), the clipped version prevents $r$ from dropping below $1-\varepsilon$: "yes, this action is bad, but don't decrease its probability too fast."

The clip creates a flat region in the objective landscape — a plateau where the gradient is zero. Once you've changed the policy "enough," the optimizer sees no slope and naturally stops. No constraint solver needed, no second-order math, no Fisher matrix. Just a min and a clip.

The clipping threshold ε controls the trade-off between stability and speed of learning. A large ε (say 0.3) allows bigger policy changes per update, learning faster but risking instability. A small ε (say 0.1) constrains updates tightly, making training safer but slower. The paper uses ε = 0.2 as the default, which works remarkably well across diverse tasks — from Atari games to simulated robotic locomotion.

Think of ε as the speed limit on a highway. Too low and traffic crawls; too high and accidents happen. The paper found that 0.2 is the sweet spot — fast enough to learn efficiently, slow enough to avoid crashes.

PPO also proposes a second variant — PPO-Penalty — that uses a KL divergence penalty instead of clipping. The idea: add a penalty term $\beta \cdot KL[\pi_{\theta_\text{old}}, \pi_\theta]$ to the objective, then adapt $\beta$ automatically:

• If the actual KL divergence after an update is too large (above $1.5 \times d_\text{targ}$), increase $\beta$ — tighten the leash.
• If the actual KL is too small (below $d_\text{targ} / 1.5$), decrease $\beta$ — loosen the leash.

This way the penalty strength self-tunes. In practice, PPO-Clip tends to be preferred because it is simpler and slightly more robust, but PPO-Penalty can be useful when you want more direct control over policy divergence.

The full PPO algorithm is beautifully simple. In each iteration:

1. Collect — run the current policy in the environment for $T$ timesteps across $N$ parallel actors, producing a batch of (state, action, reward) tuples.
2. Estimate advantages — use Generalized Advantage Estimation (GAE) to compute $\hat{A}_t$ for each timestep. GAE balances bias and variance through its $\lambda$ parameter.
3. Optimize — for $K$ epochs, shuffle the batch into minibatches and run SGD on the combined objective:

$L_t(\theta) = L_t^{CLIP}(\theta) - c_1 L_t^{VF}(\theta) + c_2 S[\pi_\theta](s_t)$

where $L^{VF}$ is the value function loss (training the critic) and $S$ is an entropy bonus that encourages exploration. 4. Repeat — set $\theta_\text{old} \leftarrow \theta$ and go to step 1.

The key insight: because the clipped objective prevents destructive updates, you can safely reuse the same batch for multiple epochs ($K = 3\text{–}10$) instead of discarding it after one gradient step. This dramatically improves sample efficiency.

The paper tested PPO on two domains:

• Continuous control (MuJoCo): simulated robots learning to walk, run, and hop. PPO matched or exceeded TRPO on almost every task, while being much simpler.
• Atari games: 49 classic Arcade games. PPO matched or exceeded the performance of A2C (a synchronous version of A3C) and ACER (an off-policy actor-critic method) on the majority of games, with a single set of hyperparameters.

The critical result: PPO achieves a favorable balance between sample complexity (how much data it needs), simplicity (how easy it is to implement), and wall-time (how fast training runs). No other algorithm at the time matched all three.

The timeline below shows how PPO became the backbone of modern AI alignment and beyond. What started as a robotics optimization trick turned into the engine behind the most consequential AI systems of the 2020s.