Progress field reinforcement learning improves vision language action control
PF-RL: Progress Field Reinforcement Learning via Goal-Conditioned Value Geometry for Vision-Language-Action Models
Robotics
Summary
Long tasks like robot manipulation are hard because rewards come only when the task finishes, making it tough to learn from the steps along the way. The authors introduce PF-RL, a way to understand and measure progress at each step by comparing current situations to the goal in a structured space. This gives the robot better, more detailed feedback to learn faster and handle complex tasks. They tested this method on simulated and real robot tasks and found it consistently helps improve performance.
What this means in practice
- •For robotics engineers: Improve robot control policies by providing stepwise progress feedback for better learning on long manipulation tasks.
- •For automation developers: Enhance the training of vision-language-action systems to complete complex goal-driven tasks more efficiently.
Authors
Yunpeng Qing, Yilun Kong, Sixu Lin, Ming Zhou, Yiming Fei, Shuang Luo, Yixiao Chi, Haoming Gu, Jingyuan Liu, Changxu Wei, Zhi Hou, Changqing Zou
Abstract
Reinforcement Fine-Tuning~(RFT) has emerged as a promising paradigm for improving Vision-Language-Action~(VLA) policies, yet sparse task-level outcomes provide limited credit for intermediate transitions, especially in long-horizon manipulation. A natural approach is to model intermediate task progress and use it as dense feedback for policy improvement. Despite their architectural differences, existing progress-aware methods commonly formulate task progress as an explicit scalar prediction, providing limited structure for modeling how intermediate observations relate to the task goal, which may hinder effective transition-level credit assignment. We introduce Progress Field Reinforcement Learning (PF-RL), which learns a structured goal-conditioned progress representation over pretrained VLA features and converts it into dense credit for policy optimization. A lightweight shared Progress Field head maps current and goal representations into a compact progress space, where geometric distance induces goal-conditioned value, while complementary temporal and goal-structure objectives shape the learned geometry. Transition-level value changes naturally yield dense progress advantages, enabling fine-grained credit assignment for both offline policy improvement and online reinforcement fine-tuning. Extensive experiments on LIBERO, RoboTwin2.0, and real-world bimanual manipulation tasks show that PF-RL consistently improves policy performance over strong supervised fine-tuning, reinforcement fine-tuning, and progress-aware baselines.