Commander executor method scales multi-human cooperative object manipulation
Scalable Multi-Human Cooperative Manipulation via Commander-Executor Coordination
RoboticsComputer Vision and Pattern Recognition
Summary
Doing tasks together that involve moving and handling objects is hard, especially when people have different roles and need to work step-by-step over time. The authors developed a two-part system where one part plans the big steps and shared goals, and the other part controls each person’s movements based on those plans. This approach helps multiple people work together better than before, even on complicated, long tasks. They tested it on various setups and showed it succeeds more often than previous methods.
What this means in practice
- •For robotics engineers: Develop controllers for teams of robots or humans to manipulate objects cooperatively with distinct roles and timed coordination.
- •For animation and game developers: Create realistic multi-character object interaction sequences where characters coordinate based on shared plans and roles.
Authors
Buzhen Huang, Xingchen Wu, Yuanwang Yang, Chenyi Xie, Kun Li
Abstract
Multi-human cooperative object manipulation is a highly challenging topic in humanoid control. While existing methods can handle simple collaborative carrying tasks, they struggle with role-dependent, long-horizon, and sequential cooperative manipulation. To address this issue, we propose a hierarchical commander-executor framework that enables multiple agents to accomplish complex collaborative tasks in distinct ways. Specifically, a commander combines instructions and scene information for joint semantic and geometric planning. It then decomposes the task into role-specific subtasks and generates shared object-centric coordination plans, specifying execution timing, target positions, and manipulation actions for each agent. Conditioned on the commander's high-level strategy and each agent's local observations, the executor learns a low-level physics-based policy through a solo-to-cooperative curriculum to realize coordinated human-object interactions. The planned object trajectory provides a shared coordination target, while root positions and grasping instructions specify role-dependent execution cues. We evaluate our method on a variety of object manipulation tasks, demonstrating its ability to execute long-horizon tasks that combine different types of collaboration and to scale from two-agent to multi-agent cooperation. Extensive experiments further show that our approach achieves higher success rates compared to state-of-the-art methods. Code will be made publicly available at https://github.com/TJU-3DV/Co-Manipulation.