Regularised reinforcement learning estimates improve with recommender regret
Recommender regret for improving regularised RL estimation with bandits and MCTS
Machine Learning
Summary
Estimating the quality of decisions during reinforcement learning often focuses on how efficiently choices are sampled, not on how good the resulting estimates are. The authors treat this estimation as a recommendation problem, introducing a new way to measure how good these recommendations are with a concept called recommender regret. They show that even simple approaches like uniform sampling can meet these measurement goals. Their new framework combines bandit algorithms and tree search methods, leading to faster and better learning in games and scientific tasks.
What this means in practice
- •For reinforcement learning engineers: Improve policy estimation speed and accuracy in reinforcement learning by integrating recommender regret principles with bandit and Monte Carlo tree search approaches.
- •For game ai developers: Use recommender MCTS to achieve faster learning and better decision-making quality in arcade and similar game environments.
Authors
Danilo Vucetic, Alexandre Larouche, Mohammad Pedramfar, Audrey Durand, Gauthier Gidel
Abstract
Regularised reinforcement learning (RL) algorithms rely on estimators, like Monte Carlo tree search or one-step look-ahead, to provide policy, value, or Q-value estimates in training and inference. However, existing bandit-based estimation frameworks evaluate sampling efficiency, not the quality of estimates. We address this by formalising estimate generation as a recommendation process, treating bandit-based sampling as an improvement operator. To evaluate the quality of a recommendation (estimate), we define regulariser-aware recommender regrets, analysing their lower bounds, and proving that simple algorithms like uniform sampling can achieve these bounds. Finally, we unify bandit- and search-based target improvement under recommender MCTS. Across synthetic trees, arcade games, and scientific discovery tasks, recommenders often yield faster target convergence and superior downstream performance. Our experiments characterise for the first time the potential of regularised MCTS in scientific applications.