Progress reward model improves robot learning on complex tasks
Progress Reward Model for Reinforcement Learning via Large Language Models
Artificial Intelligence
Summary
Reinforcement learning struggles with tasks where rewards are rare and delayed. To help, the authors created a new system called the Progress Reward Model for RL that breaks hard tasks into smaller steps and uses a progress-based reward to guide learning. This approach helps robots learn better and faster by giving them clearer feedback along the way. Experiments showed it outperformed existing methods that use large language models for planning or rewards.
What this means in practice
- •For robotics engineers: Develop control policies that learn complex, long-horizon robot tasks more efficiently using progress-based rewards.
- •For automation system integrators: Implement smarter task planning and reward systems in automated machines to improve their adaptability to complex workflows.
Authors
Xiuhui Zhang, Ning Gao, Xingyu Jiang, Yihui Chen, Yuheng Pan, Mohan Zhang, Yue Deng
Abstract
Traditional reinforcement learning (RL) algorithms face significant limitations in handling long-term tasks with sparse rewards. Recent advancements have leveraged large language models (LLMs) to enhance RL by utilizing their world knowledge for task planning and reward generation. However, planning-based approaches often depend on pre-defined skill libraries and fail to optimize low-level control policies, while reward-based methods require extensive human feedback or exhaustive searching due to the complexity of tasks. In this paper, we propose the Progress Reward Model for RL (PRM4RL), a novel framework that integrates task planning and dense reward to enhance RL. For high-level planning, a complex task is decomposed into a series of simple manageable subtasks, with a subtask-oriented, fine-grained progress function designed to monitor task execution progress. For low-level reward generation, inspired by potential-based reward shaping, we use the progress function to construct a Progress Reward Model (PRM), providing theoretically grounded optimality and convergence guarantees, thereby enabling effective policy optimization. Experimental results on robotics control tasks demonstrate that our approach outperforms both LLM-based planning and reward methods, achieving state-of-the-art performance. Code is available at https://github.com/deng-ai-lab/PRM4RL.