RobotUse system improves robot action planning and learning
RobotUse: Allocating Computation, Context, and Decisions
Robotics
Summary
Robots often struggle to connect their planned moves with what actually happens and to keep track of details when trying things again. The researchers created RobotUse, a tool where robots pick targets visually while the system handles movement and control. RobotUse keeps track of each small goal and learns from past attempts to improve future actions. It performed better than a previous system on robot tasks and can learn even with imperfect feedback in the real world.
What this means in practice
- •For industrial automation engineers: Enable robots on assembly lines to better plan and revise physical actions for improved task success without complex manual coding.
- •For robotic software developers: Build robot agents that visually select targets and update plans based on previous attempts, reducing reliance on predefined action code.
Authors
Junhoo Lee, Injun Baek, Seungyeon Kim, Suhyun Jeon, Minkyu Kim, Baekseung Kim, Nojun Kwak
Abstract
Robot agents must connect their intended actions to observed outcomes while retaining the context needed to revise their choices over repeated attempts. Existing interfaces often leave these choices inside predefined tools or require agents to manage detailed execution code and its growing history. We introduce RobotUse, a robot agent harness that organizes computation, context, and decisions around specifying and revising physical actions. Agents visually select targets and poses, while the backend handles geometry, motion planning, and control. Subagents retain detailed interactions within each subgoal and return the information needed for subsequent decisions. Continual harnessing lets agents learn from execution by updating a persistent playbook. On RoboLab, RobotUse achieves 45% task success, outperforming CaP-X by 6.7 percentage points while maintaining compact decision contexts and reducing reliance on predefined action abstractions. Furthermore, we show that RobotUse learns from real-world execution despite imperfect feedback and transfers what it learns to subsequent tasks. Project page is available at https://robotuse-team.github.io/.