Ensemble method improves online algorithm choice under data limits
Uncertainty-Aware Selection of Online Algorithms with Simulator Ensembles
Machine LearningArtificial Intelligence
Summary
Choosing the best reinforcement learning algorithm often involves simulating different options before trying them out. When there’s limited data, picking the best algorithm from a single simulator can be unreliable. The authors explore using many slightly different simulators together to better capture uncertainty and improve the selection. They show this method reduces mistakes theoretically and works well in robot control tests, making online learning more dependable.
What this means in practice
- •For robotics engineers: Improve tuning of reinforcement learning algorithms in robot control tasks by using ensemble simulators to select more reliable reward shaping parameters.
- •For machine learning platform teams: Enhance automated online algorithm selection in reinforcement learning pipelines by integrating uncertainty-aware ensembles to handle limited offline data.
Authors
Yongyi Guo, Zifan Xu, Ziping Xu, Kelly W. Zhang
Abstract
The performance of online reinforcement learning depends critically on design choices, especially those that affect exploration. These choices are often selected by fitting a simulator to offline data, evaluating candidate algorithms in that simulator, and deploying the best-performing one. The simplest Plug-In selection rule simply selects the best performing algorithm on the fitted simulator, making evaluations unreliable when the offline data used to fit the simulator are limited. We investigate Uncertainty-Aware selection, which forms an ensemble of simulators---for example, obtained by bootstrap resampling---and selects the online algorithm with the best average performance across the ensemble. While ensemble-based approaches have been used to mitigate distribution shift and facilitate sim-to-real transfer, we formally show that this approach can mitigate the effects of limited data when fitting the simulator and theoretically has significant regret gains compared to Plug-In selection in multi-armed bandits. We also empirically investigate the Uncertainty-Aware selection approach in deep RL experiments on robotic control tasks that involve selecting reward-shaping hyperparameters, and show that it leads to more reliable selection and improved online performance.