Semi-gradient td learning shows stable convergence under entropy regularization
On the Convergence of Semi-Gradient TD: A Mean Field Perspective
Machine LearningArtificial Intelligence
Summary
Temporal-difference learning is a technique used to help computers learn by trial and error, but it can sometimes behave unpredictably when using neural networks. The authors show that adding a bit of randomness, or 'entropy,' helps stabilize this learning process, making it reliably find a solution. They prove mathematically that this approach converges quickly to a stable point and also explain how this works even when the computer is learning from a different strategy than the one it follows. This helps make reinforcement learning methods more reliable when using complex neural networks.
What this means in practice
- •For ai engineers: Ensure stable training of reinforcement learning agents using neural networks with entropy regularization.
- •For robotics teams: Design off-policy control algorithms that reliably converge by integrating entropy-based smoothing in learning updates.
A theory result. No direct application yet.
Authors
Haruki Settai, Tadashi Kozuno, Shinji Ito
Abstract
Semi-gradient temporal-difference learning is a core algorithmic component of reinforcement learning, but bootstrapping with function approximation generally induces a non-conservative vector field and can lead to instability. We analyze this effect for two-layer neural networks in the mean-field regime and identify a simple stability mechanism in which entropy-induced relaxation dominates the sensitivity of the moving bootstrap target. Under this condition, we prove exponential convergence to a unique fixed point and quantify its bias relative to the unregularized Bellman solution. We also show that the same mechanism extends to entropy-regularized off-policy control through soft Q-learning.