Mixing policy improves average reward q learning convergence and exploration
Mixing Regularization for Average-Reward $Q$-Learning and $Q$-Planning
Machine LearningArtificial Intelligence
Summary
Learning the best actions in situations where rewards accumulate over time is hard when the environment only slowly mixes or changes. The authors study how having a known policy that mixes well helps guide learning new policies by nudging it towards this known one. They analyze how this regularization affects convergence speed and exploration, proving new guarantees and showing better results especially when environments have difficult bottlenecks. Their theory also explains why some ways of regularization work better than others.
What this means in practice
- •For robotic system developers: Improve learning efficiency and exploration by integrating known mixing behaviors into reinforcement learning policies in average reward tasks.
- •For industrial control engineers: Design control policies with enhanced stability and sample complexity by using mixing-based regularization in average reward environments.
Authors
Julien Roger, Sam Bodmer, Jian Qian, Benjamin Riviere
Abstract
The average-reward setting for reinforcement learning is challenging because its Bellman operator is known to contract in span only when the underlying Markov chain is mixing. Whereas existing work typically assumes a contraction property, e.g., through environment mixing, we consider the setting where we are given a reward-agnostic $m$-step mixing policy, and we control the contraction and exploration properties of the learned policy by regularizing toward the mixing policy. We provide finite-time analysis of $Q$-learning algorithms that can be instantiated with multiple choices of regularization, with guarantees measured with respect to the regularized fixed point. Our framework exposes novel regularizer-dependent objects that control contraction, exploration, and operator bias. In the one-step setting, our rates match the best known rates for $Q$-iteration and synchronous $Q$-learning, and for $m$-step mixing we derive new rates. We also study $Q$-planning variants, and show that planning acts as time-localized learning, where the planning horizon $H$ trades off the bias from the boundary value against the sample complexity. Finally, we present numerical studies validating the analysis: (i) knowledge of a strong mixing policy improves convergence and exploration, and its advantage increases with bottleneck severity and, (ii) our theory predicts the convergence and bias rankings of regularizers from the $α$-divergence family.