Aligning robot rewards with task progress improves learning efficiency
PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning
RoboticsArtificial Intelligence
Summary
Teaching robots to do tasks often involves showing them examples or giving them feedback. Examples give broad guidance but might miss details, while feedback can be very precise but is sparse. The authors created a method called PHIRL that combines both, using information about how far along a task is to better teach robots what they should aim for. This approach needs fewer detailed examples and helps robots learn safer and more reliable behaviors. They tested PHIRL with real and simulated robots and found it worked much better than previous methods.
What this means in practice
- •For robotics engineers: Train robots to complete tasks more efficiently by combining human examples with progress feedback for better reward design.
- •For automation system designers: Improve task success rates in automated systems by using progress-informed rewards that resist exploitation.
Authors
Hang Yu, James Staley, Cheng Xi Tsou, Xiujin Liu, Wenchang Gao, Jindan Huang, Shijie Fang, Zhegong Shangguan, Angelo Cangelosi, Reuben Aronson, Elaine Short
Abstract
Human demonstrations provide dense policy-level information but sometimes lack local precision. Human feedback presents accurate local critiques, but offers sparse evaluations rather than direct policy guidance. We propose Progress-Heuristicized Inverse Reinforcement Learning (PHIRL), a data-efficient framework that learns robust reward functions by jointly leveraging demonstrations and feedback. Specifically, we use progress, a feedback modality that describes cumulative task completion. PHIRL iteratively infers a reward function from demonstrations via inverse reinforcement learning, calculates the learned rewards over the progress-annotated demonstrations, and aligns the rewards with progress annotations over four dimensions. We evaluate PHIRL on real and simulated robot tasks, with additional exploration using a fine-tuned vision-language model to provide progress feedback. Results demonstrate that PHIRL significantly outperforms the baselines, achieving substantially higher environmental return rewards and task success with only twenty percent of demonstrations annotated. Analysis of reward-hacking scenarios demonstrates that PHIRL learned reward functions are reliable against exploitation.