Understanding and Stabilizing Deep Q-Learning via Controlled Bootstrapping and Regulated Value Dynamics
2026-08-17 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors studied why training deep Q-learning models can be unstable. They looked at three parts causing this: errors in the value updates, sensitivity when choosing actions, and problems when reusing data too much. They found a pattern where certain rewards make the system get stuck in a bad cycle and identified sudden big changes in model parameters. To fix these, they suggest ways to control learning steps, use multiple estimates, and regulate parameter spikes, showing better results on game benchmarks.
Deep Q-learningReinforcement LearningBellman EquationBootstrappingOverestimation BiasAction SelectionRegression NoiseParameter DynamicsAtari-100KProcgen
Authors
Bozhou Chen, Yongyi Wang, Hanyu Liu, Xionghui Yang, Wenxin Li
Abstract
Deep Q-learning (DQL) has achieved remarkable empirical success in reinforcement learning, yet its training process remains notoriously unstable. Existing studies often attribute instability to isolated factors such as overestimation bias or representation learning issues, lacking a unified understanding of how different sources of instability interact during recursive value estimation. In this work, we provide a systematic analysis of instability in deep Q-learning from three complementary perspectives: operator-level bias in Bellman bootstrapping, estimator-level sensitivity of greedy action selection to regression noise, and parameter-dynamics imbalance under aggressive data reuse. We identify a reward-triggered self-reinforcing trap and characteristic parameter spike dynamics, then derive stabilization principles for controlled bootstrapping, ensemble quantile estimation, and spike-based parameter regulation. Experiments on Atari-100K and Procgen demonstrate competitive performance and improved training stability.