Grana improves robot reward tracking for long tasks
Grana: Compositional Reward Modeling for Long-Horizon Robot Manipulation
Robotics
Summary
Robots need to know how well they are doing at complex tasks that take a long time and have many steps. The authors present Grana, a system that helps robots understand how much progress they’ve made on subtasks described in plain language and combine that to see the overall progress. They trained Grana on many short tasks to predict progress even on longer, more complex ones without extra training. This helps robots learn better and work longer on complicated tasks by tracking their progress more accurately.
What this means in practice
- •For robotics engineers: Design robot control systems that estimate progress on complex manipulation tasks with natural language subtasks to improve long-horizon task performance.
- •For industrial automation teams: Deploy more adaptable robotic systems capable of tracking incomplete steps in varied assembly or packaging workflows without fixed action sets.
Authors
Zi-ang Cao, Kaiyuan Zheng, Yinzhen Xu, Haonan Duan, Jiasheng Gu, Fengyuan Hu, Nan Zhu, Kun Lei, Kevin Lin, Ruijie Zheng, Shenyuan Gao, Fangqi Zhu, Zhe Zhang, Radha Poovendran, Linxi Fan, Guanzhi Wang, Yuke Zhu
Abstract
Robot reward modeling faces two challenges: generalizing across tasks and accurately tracking progress in long-horizon manipulation. General-purpose reward models demonstrate broad task transfer, but transfer from short-task pretraining to long-horizon tasks remains underexplored. Long-horizon progress models often rely on task-specific subtask classifiers or fixed action vocabularies, limiting the reuse of learned progress knowledge when adapting to new tasks. To address these challenges, we introduce Grana, a vision-language robot reward model that learns to identify unfinished subtasks and estimate their continuous progress using open-vocabulary subtask descriptions. We train a language-conditioned model to estimate progress in short-horizon tasks and individual subtasks of long-horizon tasks. For long-horizon tasks, we combine the selected subtask's predicted progress with credit for preceding completed subtasks to estimate whole-task progress. General progress prediction requires training the model across diverse tasks with progress supervision that accounts for imperfect execution. Elapsed-time proxies can misrepresent progress when relative subtask durations vary, or failures reverse local progress. We therefore introduce an annotation protocol that converts sparse subtask boundaries and action-quality annotations into dense progress targets, without requiring annotators to assign numerical progress scores. Pretraining with this supervision on 200 diverse, mostly short-horizon manipulation tasks enables zero-shot reward prediction on unseen tasks and improves progress prediction after adaptation to multi-minute, long-horizon tasks. Experiments demonstrate (i) generalizable reward prediction, including adaptation to five long-horizon tasks, (ii) benefits from pretraining and composing subtask progress estimates, and (iii) improved downstream policy performance.