SAC policy tweaks fail to improve extreme action control performance
Unthrottling the Tanh Jacobian in SAC: A Negative Result on Bang-Bang Control and MetaDrive
Machine LearningRobotics
Summary
Soft Actor-Critic (SAC) uses a mathematical function called tanh that limits actions, which some thought might stop the system from learning well when extreme actions like full braking or full throttle are needed. The authors tried adding a simple fix to recover lost learning signals at those extreme points. But their experiments showed this fix either made things worse or didn’t help, meaning the original problem of losing gradient signals at extreme actions isn’t as straightforward as it seems. Basically, just forcing the policy to saturate at action limits doesn’t solve the challenge of controlling systems optimally at those extremes.
What this means in practice
- •For reinforcement learning developers: Avoid adding naive gradient bypasses in SAC policies when optimizing tasks requiring extreme actions, as they may harm performance.
- •For simulation platform engineers: Recognize limits of simple action-gradient fixes when tuning policy learning in driving simulators like MetaDrive to better handle hard control tasks.
Authors
Faiq Shamass
Abstract
Soft Actor-Critic (SAC) represents a continuous policy as an unbounded Gaussian that is squashed by tanh. The Jacobian of that map is $\partial a/\partial u = 1-a^2$, which vanishes as $|a|\to 1$. A natural concern is that this throttle starves the actor of critic signal exactly where extreme actions (full brake, full throttle) are optimal. We test a minimal intervention that restores the missing signal: one extra term in the actor loss whose gradient on the pre-tanh mean is the detached action-gradient of $Q$, with no gain parameter. On a minimum-time double integrator whose optimum is bang-bang at the action bounds, vanilla SAC already reaches near-optimal return ($-31.6$ vs. a calibrated optimum of $-30.3$) across ten paired seeds. An ungated bypass does saturate the policy (99% of eval steps with $|a|\ge 0.9$) and collapses return to $-195.5$. A gated bypass that fires only on the flat shoulder $|a|\in[0.9,0.999]$ also fails, and does so without leaving a saturated policy. Warm-started MetaDrive fine-tuning shows the same pattern: the bypass does not improve return, and where collision rate falls it is typically traded for out-of-road departures. Auto-tuned entropy coefficient rises against the bypass, which is a push toward the tails. The Jacobian effect is real. Treating it as a bug to be undone is not free, and on the tasks studied here it is not helpful. Saturating a bound is not the same as solving a problem whose optimum lives on that bound.