Physical relations improve credit assignment in embodied reinforcement learning
PRICE the Action Chunks: Physical Relational Credit Assignment for Embodied Reinforcement Learning
Robotics
Summary
Some computer programs learn to do tasks by trying actions and only finding out success or failure at the end, which can unfairly blame helpful early actions for failure. The authors developed a way to look at similarities between different attempts to better judge which parts of the action helped progress, without needing extra teaching or complex models. Their method, called PRICE, uses a graph to compare moments in different attempts and refines feedback based on these comparisons, leading to better and faster learning on robots and simulations. This helps robots learn complex tasks from just knowing if they succeeded or failed at the end.
What this means in practice
- •For robotic system developers: Improve robotic task learning speed and success rates by refining action-level feedback from final outcomes without extra supervision.
- •For industrial automation teams: Develop more efficient training methods for embodied machines using terminal success signals enhanced with physical relation comparisons.
Authors
Yangang Zou, Jiajun Lu, Weitao Zhou, Haibao Yu, Bozhou Zhang, Jiawei Wang, Honglong Tian, Minglei Li, Li Zhang
Abstract
Outcome-based reinforcement learning (RL) post-trains vision--language--action policies using terminal success signals, but assigns the same trajectory-level advantage to every action chunk. A failed episode can thus penalize useful early actions as if they caused the failure. Existing approaches seek finer-grained feedback through learned evaluators, adding task-specific supervision or additional model training. We explore, for the first time to our knowledge, whether physical relations across trajectories can provide action-chunk credit in embodied RL from terminal outcomes alone, without an auxiliary evaluator. The key insight is that rollouts reaching corresponding physical situations can serve as references for one another: their terminal outcomes provide evidence for assessing local progress. We introduce Physical Relations for Inferring Credit from Episodes(PRICE), with two components: (i) a physical relational graph that pools current and historical outcomes at corresponding chunk boundaries to estimate success potentials; and (ii) confidence-gated credit assignment that uses changes in these potentials to refine trajectory-level supervision. Our analysis connects oracle potential changes to the terminal-success objective and provides a finite-sample directional bound for outcome-independent evidence pools. Independent continuation tests show that PRICE's retained credits align with local progress, while experiments on LIBERO, RoboTwin 2.0, and real robots demonstrate improved task success over outcome-based baselines and faster learning.