SeeQ improves robot manipulation by learning subtask value functions
SeeQ: Training Generalist Value Functions for Long-Horizon Robotic Manipulation
Robotics
Summary
Robotic tasks that take many steps and require trying several times are hard for robots to do well. The authors created SeeQ, which helps robots understand the value of the specific smaller part of a task they are working on instead of the whole long task at once. This makes it easier and faster for the robot to learn from past experiences. SeeQ also teaches the robot to recognize what subtask it's doing using natural language before deciding the best action. Tested on real robots, SeeQ helps them make better choices during complex tasks.
What this means in practice
- •For robotics engineers: Design and train robot controllers that use subtasks to improve performance on complex, multi-step manipulation tasks.
- •For robotics product developers: Enhance bimanual robot products by integrating Q-function architectures that enable long-horizon task success through subtask-aware value learning.$Commercial implications: Enables commercially viable robot systems that reliably complete multi-stage tasks used in manufacturing or service robots.
Authors
Saksham Singh, Zheyuan Hu, Max Sobol Mark, Jeffrey Yu, Zackory Erickson, Aviral Kumar
Abstract
Despite rapid progress, generalist robot policies remain brittle on complex, long-horizon tasks that comprise multiple stages or require repeated attempts and deliberation on the same underlying stage before success. Q-value functions can improve these policies by ranking candidate actions or guiding policy improvement, but learning from sparse task-level rewards entails long credit-assignment horizons, difficult Bellman backups, and broad data-coverage requirements. We introduce SeeQ (Subtask-elicited Q-functions), which instead learns Q-values for the currently active subtask. This shortens the value-prediction horizon and enables effective learning with temporal-difference (TD) objectives. During training, subtask-level annotations present in offline robot data provide the decomposition and enable learning from broad, potentially suboptimal robot datasets. To eliminate the need for human annotations or modular subtask prediction systems at test time, our Q-function architecture is trained to autoregressively predict the active subtask in natural language before estimating its value. We instantiate SeeQ using a base vision-language backbone, pretrain it on diverse open-source robot manipulation data, and finetune it on downstream tasks. Across four real-world manipulation tasks on two bimanual robot platforms, the SeeQ value function substantially improves best-of-N policy steering.