Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

By 2018 the deep RL community had two families of algorithms for continuous control, each with a critical weakness.

On-policy methods like PPO collect a batch of trajectories, update the policy once, and throw away the data. They are stable because the data distribution always matches the current policy, but this stability comes at an enormous cost: millions of environment steps to solve even moderately complex tasks. For a real robot, that means days of physical interaction.

Off-policy methods like DDPG store all past transitions in a replay buffer and reuse them many times. This is far more sample efficient. But DDPG learns a deterministic policy and adds noise only for exploration. This makes it brittle: the Q-function can overestimate values, the deterministic policy can exploit those errors, and the whole system can diverge with a bad hyperparameter choice or unlucky random seed.

The field needed a method that inherits the sample efficiency of off-policy learning while achieving the stability that on-policy methods are known for.

Standard RL seeks a policy that maximizes expected cumulative reward. Maximum entropy RL adds a twist: the agent also earns a bonus proportional to the entropy of its policy — a mathematical measure of how spread out or unpredictable its actions are.

Think of it as the difference between a chess player who only knows one opening and a grandmaster who has mastered many. The grandmaster's repertoire is broad (high entropy), which makes them harder to predict and more adaptable. The entropy bonus rewards this breadth of behavior.

Concretely, instead of maximizing just the return, the agent maximizes the soft return — the sum of rewards plus a scaled entropy bonus at each step. The scaling factor α (called the temperature) controls the trade-off: high α means more exploration, low α means more exploitation. This single idea is the foundation of everything SAC does.

Why does adding entropy help? Three reasons. First, it encourages exploration: a policy with high entropy visits more states, discovering rewards that a narrow policy would miss. Second, it prevents premature convergence: the policy cannot collapse to a single action, so it remains flexible. Third, it provides robustness: because the policy maintains multiple good strategies, small changes in the environment don't cause catastrophic failure.

In standard RL, the state-value function V^π(s) measures the expected cumulative reward from state s under policy π. The Q-function Q^π(s,a) measures the same but conditioned on taking action a first. The Bellman equation links them recursively.

In maximum entropy RL, we redefine these to include entropy. The soft state-value function includes an entropy bonus at each step. The soft Q-function excludes the entropy from the first step (since the action is already chosen) but includes it for all future steps. This gives us the soft Bellman equation, the recursive backbone of SAC.

The intuition is simple: V^π now measures not just "how much reward will I get" but "how much reward will I get while staying as unpredictable as possible." A state is more valuable if the policy can act diversely from it.

SAC trains three neural networks simultaneously, like a team where each member has a distinct role.

The Actor (Policy Network π_θ) takes a state and outputs a probability distribution over actions. Specifically, it outputs the mean μ and log-standard-deviation log σ of a Gaussian distribution. A sample from this Gaussian is then squashed through a tanh function to keep actions within bounds. This is the "stochastic actor" in the paper's title.

Two Critics (Q-Networks Q_φ₁ and Q_φ₂) each take a state-action pair and output a scalar Q-value. Using two critics and taking their minimum is called the clipped double-Q trick — it prevents the Q-functions from overestimating, which was a persistent problem in DDPG.

Target Networks (Q_φ̄₁ and Q_φ̄₂) are slowly-updated copies of the critics. Instead of copying weights periodically (hard update), SAC uses Polyak averaging (soft update): each step, the target weights move a tiny fraction toward the main weights. This acts like a low-pass filter, smoothing out noisy Q estimates and stabilizing training.

SAC needs a policy that is stochastic (for entropy), bounded (physical actuators have limits), and differentiable (for gradient-based learning). The squashed Gaussian achieves all three.

The actor network outputs μ(s) and σ(s) — the mean and standard deviation of a Gaussian. A noise sample ξ is drawn from a standard normal, and the raw action is computed as μ(s) + σ(s) ⊙ ξ. This is the reparameterization trick: it expresses the random action as a deterministic function of the network parameters plus independent noise, making it possible to backpropagate through the sampling operation.

The raw action is then squashed through tanh, which maps it to the interval (−1, 1). This bounding is essential for physical systems: a robot joint can't rotate infinitely. The log-probability of the squashed action requires a correction term (the log-determinant of the Jacobian of tanh), which the paper derives in its appendix.

SAC's training loop has three stages that repeat at each gradient step.

Stage 1 — Update the Critics. Sample a mini-batch from the replay buffer. For each transition (s, a, r, s', d), compute a target Q-value using the target networks. The target includes the entropy term: it is the reward plus γ times the minimum of the two target Q-values at the next state, minus α times the log-probability of the next action. Each critic is trained to minimize the mean squared error between its prediction and this target.

Stage 2 — Update the Actor. The actor's goal is to maximize the expected Q-value (using the minimum of the two critics) minus α times the log-probability. This encourages the actor to find actions that are both high-value and high-entropy. The reparameterization trick makes this gradient computable.

Stage 3 — Update the Target Networks. The target network weights are slowly moved toward the main critic weights via Polyak averaging. A typical Polyak coefficient ρ is 0.995, meaning only 0.5% of the new weights bleed in per step.

A single Q-network tends to overestimate values, because the maximum of noisy estimates is biased upward. This is like asking two biased judges for scores and always taking the higher one — you systematically overrate the performance.

SAC (borrowing from TD3) trains two independent Q-networks and uses the minimum of their predictions as the target. Think of it as asking two critics for their opinion and trusting the more conservative one. This "pessimistic" estimate counteracts overestimation without adding significant computational cost. Each critic sees different mini-batches and develops slightly different biases, so their minimum is a more honest estimate.

The original SAC paper treats α as a fixed hyperparameter. But choosing the right temperature is tricky: too high and the policy is too random to learn; too low and it loses the entropy benefits. The follow-up paper (Haarnoja et al., 2018b) solved this elegantly by making α learnable.

The idea: impose a constraint that the policy's entropy must stay above a target value H̄ (typically set to −dim(A), the negative of the action dimensionality). Then use dual gradient descent to learn α: if entropy drops below the target, α increases to encourage more randomness; if entropy is already high enough, α decreases to let the policy focus on reward. This automatic adjustment eliminates the most sensitive hyperparameter and makes SAC work across diverse environments without tuning.

vs DDPG: Both are off-policy actor-critic algorithms using a replay buffer. But DDPG uses a deterministic policy and adds external noise for exploration, making it fragile. SAC uses a stochastic policy with built-in entropy maximization — exploration is not an afterthought but a first-class objective. SAC also uses double-Q learning to prevent the overestimation that plagues DDPG.

vs TD3: TD3 introduced the clipped double-Q trick and delayed policy updates to fix DDPG's instability. SAC incorporates the double-Q trick but takes a different approach to stability: instead of delayed updates, it uses entropy regularization and a stochastic policy. SAC was published roughly concurrently with TD3 and the two represent complementary solutions to DDPG's problems.

vs PPO: PPO is on-policy — it discards data after each update, requiring far more environment interactions. It is stable because the policy changes slowly (clipped surrogate objective), but this stability comes at the cost of sample efficiency. SAC achieves comparable or better stability while being dramatically more sample efficient through off-policy learning.

SAC's legacy extends beyond a single algorithm. It demonstrated that entropy regularization — the idea of rewarding behavioral diversity — is a general principle applicable to offline RL, model-based RL, multi-task RL, and robotics. The maximum entropy framework that SAC made practical has become a standard tool in the deep RL arsenal.