Large scale study reveals best offline policy learning methods vary
JumpStart Your Policy Learning with Lessons from 160,000 Training Runs
Machine Learning
Summary
Offline policy learning helps machines learn from past data to make decisions without exploring new actions. The authors found that which method works best depends a lot on how you tune settings and the data you use. They trained over 160,000 policies on many different tasks and saw no single method won everywhere. They also created a tool to recommend the best method for a given task based on this knowledge. Their work and released data aim to make future progress in this area more trustworthy and easier.
What this means in practice
- •For machine learning engineers: Select effective offline learning algorithms and settings tailored to specific tasks using a data-driven recommender from large-scale training results.
- •For robotics developers: Improve robot decision-making by choosing well-tuned offline policies suited for different environments using insights from extensive algorithm comparisons.
Authors
Nabil Omi, Eric Bae, Chung Yik Edward Yeung, Siddhartha Sen, Ali Farhadi
Abstract
Reliable progress in offline policy learning depends on careful reporting, well-tuned baselines, and evaluation across diverse conditions. Prior work has shown that results can be sensitive to reporting choices, hyperparameter tuning, and dataset properties, but these sources of variability have not been systematically investigated together at the scale needed to understand how they shape conclusions. To address this gap, we present a large-scale empirical study of offline reinforcement and imitation learning, training over 160,000 policies across 114 datasets. At this scale, no algorithm dominates: aggregate performance among the strongest methods is often close, but the leaders differ substantially across environments. We find that proper hyperparameter tuning frequently reshuffles perceived algorithm rankings and that benchmark composition can produce conflicting conclusions. We also study hyperparameter sensitivity and transfer across environments, identifying a simple strategy for deriving strong default configurations. We use our findings to develop a dataset-conditioned recommender that provides task-specific algorithm recommendations for practitioners. Finally, we release JumpStart: a resource suite containing every trained policy, per-model scores and hyperparameters, strong baselines across all environments, training and evaluation code, and an extensible website for retrieving, analyzing, and contributing results. Together, these resources aim to make offline policy-learning research more reliable and enable future work beyond the scope of this study.