Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learning
2026-08-03 • Machine Learning
Machine Learning
AI summaryⓘ
The authors address a problem in reinforcement learning where using multiple future steps to update values can cause pessimistic bias due to linking decisions to bad actions later on. They propose a new method called Expectile n-step Q-learning (ENQ), which uses an asymmetric loss function to reduce this bias while keeping the math stable. Their theoretical analysis shows ENQ behaves well in both deterministic and random environments. In experiments across many tasks, ENQ performed competitively with other methods and was more efficient in training. The authors also showed it benefits more from using multiple critics to improve learning.
Reinforcement LearningOff-policy LearningMulti-step ReturnsTemporal-Difference LearningQ-learningExpectile LossBias-Variance TradeoffDeterministic DynamicsStochastic DynamicsEnsemble Methods
Authors
Abdelghani Ghanem, Mounir Ghogho
Abstract
Multi-step returns accelerate reward propagation in off-policy reinforcement learning, but couple the evaluation of each decision to the suboptimal logged actions that follow it, inducing a pessimistic bias that grows with the horizon. We propose Expectile $n$-step Q-learning (ENQ), which replaces the symmetric $n$-step temporal-difference (TD) loss with an asymmetric expectile loss on the action-value error, with expectile level $τ$ as the only method-specific hyperparameter added beyond $n$-step TD. We prove that the ENQ operator is a $γ^{n}$-contraction. Under deterministic dynamics, at $τ=1$, its bias vanishes at the optimal action-value function $Q^*$ on covered in-support pairs, and the corresponding fixed point satisfies the separation-$n$ instance and its multiples of the lower-bound inequality used by Long-Horizon Q-learning (LQL). Under stochastic dynamics, the operator bias admits two-sided bounds with horizon-independent noise constants. Using a single expectile level $τ=0.8$ and a fixed backup horizon across 27 manipulation and navigation task instances, ENQ is competitive with LQL on aggregate, achieves higher measured training-step throughput in our profiling study, and benefits more from a ten-critic ensemble in a controlled scaling experiment.