Q learning adapts planning horizons using entropy guided mixing
Q($γ_H$): Entropy-Guided Horizon Mixing in Multi-Discount Q-Learning
Machine Learning
Summary
Q-learning normally uses a fixed planning horizon, which can be too short for learning tasks that need long-term thinking and too long for early exploration. The authors propose Q(γ_H), which keeps multiple horizons for decision making and mixes them adaptively based on how confident the model is, using a measure related to entropy. This helps the learning system gradually shift from focusing on short-term rewards to long-term rewards as it gains more certainty. Tests show this approach works better on tasks where rewards are delayed, without needing to change the usual learning steps or guess the best fixed horizon.
What this means in practice
- •For robotics developers: Adjust control strategies in robots to balance short-term exploration with long-term goal achievement using adaptive horizon mixing in reinforcement learning.
- •For game ai engineers: Improve learning efficiency in game agents by dynamically combining multiple planning horizons based on confidence during training.
Authors
Arwa Albelaihi, Abdulrahman Altahhan
Abstract
A fixed discount factor in Q-learning imposes a single planning horizon throughout training, despite the differing demands of early exploration and long-term credit assignment. We propose Q($γ_H$), a multi-horizon Q-learning framework that maintains multiple discount-specific heads and adaptively combines them at control time using an entropy-derived confidence signal. The resulting state-dependent mixing weights shift control from shorter- to longer-horizon heads as policy confidence increases, enabling adaptive horizon mixing without modifying individual Bellman updates. Across classical control and deep reinforcement-learning benchmarks, Q($γ_H$) shows its strongest improvements on delayed-credit tasks relative to strong fixed-discount baselines. By decoupling horizon adaptation from the underlying learning updates, Q($γ_H$) provides a lightweight framework for adaptive planning-horizon control in reinforcement learning, avoiding the need for manual discount-factor selection while remaining compatible with standard value-learning methods.