Robot rewards improve learning by tracking task progress relations
Event2Reward: Learning Robotic Rewards from Progress Relations
RoboticsMachine Learning
Summary
Teaching robots to understand how they are making progress during tasks can help them learn better. The authors noticed that sometimes robots get stuck or move backward in a task, so just tracking time doesn’t tell the full story. They introduced a way to learn from both absolute progress and how progress changes, even when exact progress is unknown. This method helps robots learn more efficiently, especially when they see examples of failures and recoveries. Their approach led to big improvements in robot performance on many tasks both in simulation and on real machines.
What this means in practice
- •For robotics engineers: Train robots to perform complex tasks more effectively by learning detailed progress rewards from demonstrations including failures.
- •For automation system developers: Improve robotic task automation by enhancing reward signals to better guide learning during stalled or regressing task phases.
Authors
Junnan Nie, Zitian Fan, Jiachen Zhang, Junyi Lao, Likui Zhang, Chenghao Liu, Xiaoyang Lu, Yunbo Wang, Yixue Zhang, Linbo Zhai, Jiayi Li, Mengzhe Wang, Xichang Wu, Bin Wang, Chang Liu, Tianle Zhang, Songfang Huang
Abstract
Learning task progress from robot demonstrations provides a practical way to construct dense shaping rewards for robotic reinforcement learning. A common approach uses normalized demonstration time as progress labels, implicitly assuming monotonically increasing task progress. During policy exploration, however, actual task progress can stall or even regress. Such models may mistake regressions for progress, assigning positive shaping rewards to actions that cause progress loss. Real suboptimal and failed executions offer corrective experience, but assigning absolute progress values to their states is difficult. Our key observation is that stalls, regressions, and recoveries establish local relative progress relations even when absolute progress is unknown. We introduce Event2Reward, which uses normalized temporal labels to anchor the progress scale and these within-trajectory relations to constrain local changes, combining absolute and relative progress supervision. Its progress predictions serve as potentials for reward shaping. We curate an event-annotated dataset from approximately 100K raw real-robot execution records, covering successful, suboptimal, and failed executions, without requiring framewise absolute progress labels. Under the same training budgets, Event2Reward improves average policy success by 18.16 percentage points over a baseline using only sparse task rewards across 40 tasks in four LIBERO suites. On real robots, average success after 30 minutes of wall-clock training reaches 60.0% on three precise insertion tasks and 90.0% on three object manipulation tasks, exceeding the sparse-reward baseline under the same budget by 36.7 and 23.3 percentage points, respectively. Ablation studies further show that absolute progress supervision and relative progress relations are complementary, with joint supervision yielding better downstream policy learning.