Reinforcement learning method improves value evaluation over long horizons
Limiting-Kernel Q($λ$): Bridging Short and Long Horizons
Machine Learning
Summary
Evaluating future rewards accurately helps computers learn better strategies over time. The authors developed a new technique called Limiting-Kernel Q(lambda), which combines short-term and long-term predictions efficiently. This method keeps computations manageable while improving learning for tasks that need long-term planning. Their tests show this approach works better than existing ones, especially in tasks requiring extended foresight.
What this means in practice
- •For robotics engineers: Improve long-horizon decision making in robot control tasks with efficient value estimation.
- •For autonomous vehicle teams: Enhance policy learning for continuous driving tasks that require planning over extended time frames.
Authors
Tolga Ok, Arman Sharifi Kolarijani, Peyman Mohajerin Esfahani, Mohamad Amin Sharifi Kolarijani
Abstract
In value-based reinforcement learning, improving the accuracy of policy evaluation has been shown to improve downstream policy optimization performance. The widely adopted family of approximations relying on $n$-step truncation yields computationally efficient value estimators but is inherently limited to a short evaluation horizon. In contrast, methods that exploit the global structure of the transition dynamics can accelerate policy evaluation, but their memory and computational requirements often limit scalability to large or continuous state spaces. To reconcile these limitations, we introduce Limiting-Kernel Q($λ$) (LKQL), an off-policy value estimator that combines $n$-step truncation with a long-horizon approximation based on the limiting kernel (LK). LKQL has the same order of complexity as $n$-step estimators and integrates directly into both on- and off-policy actor-critic algorithms. We prove that, under aperiodicity and in the near-on-policy regime, the operator underlying LKQL improves the policy evaluation convergence rate over its truncated counterpart for sufficiently large $n$, and that LKQL itself converges almost surely to the optimal values in finite Markov decision processes (MDPs) under a fixed behavior policy. On the MuJoCo continuous-control benchmark, we show that LKQL improves over $n$-step baselines in most settings, particularly on long-horizon tasks.