Accelerating Q-learning through Efficient Value-Sharing across Actions
2026-06-29 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors focus on improving how computers learn the value of different actions in tasks like video games. Normally, learning these values can be slow because each action is learned separately and starts near zero. They introduce a new method called the mean-expansion layer that helps share information between actions in the same situation, making learning faster and more accurate. When added to existing deep learning models, this layer helps the models perform better across many games and reduces overestimation of values.
Action-valuesQ-learningReinforcement LearningDeep Q-networksImplicit Quantile NetworksMean-expansion layerValue overestimationAction gapsState-action pairs
Authors
Prabhat Nagarajan, Brett Daley, Martha White, Marlos C. Machado
Abstract
Action-values are foundational to many control algorithms such as Q-learning. Therefore learning action-values efficiently is central to reinforcement learning (RL). However, learning them can be slow, requiring many updates to move values from their initialization, typically near zero, to their true values, which may be far from zero. Moreover, action-value learning algorithms typically update each state-action pair independently, without learning shared value structure across actions within a state. In this paper, we address these inefficiencies by introducing the mean-expansion layer, which accelerates action-value learning by sharing values across actions within a state and by changing the problem from directly learning potentially large action-values to learning a lower-norm representation of them. In deep RL, this layer can be applied as a parameter-free addition to Q-network architectures without altering the underlying algorithm. Applied to deep Q-networks and implicit quantile networks, it improves aggregate performance across 57 Atari games while increasing action gaps and dramatically reducing value overestimation.