Hierarchical framework enables flexible multi-agent object manipulation
From Solo to Ensemble: A Hierarchical Framework for Composable Multi-Agent Human-Object Interaction
Robotics
Summary
Manipulating objects with multiple people working together is hard because their movements and interactions need to be coordinated. The authors created a system that takes skills learned by one person handling an object and turns them into reusable building blocks. These blocks let multiple agents work together by planning coordinated movements of the objects instead of controlling every detail of human movement. Their experiments show this helps teams handle different objects and tasks more reliably.
What this means in practice
- •For robotic manipulation engineers: Design multi-robot systems that coordinate object handling by reusing single-robot motion skills to improve adaptability across objects and tasks.
- •For animation software developers: Create realistic multi-character interaction animations by coordinating object-centered proxy motions instead of full-body control.
Authors
Zekai Deng, Kangyi Chen, Ye Shi, Jingya Wang
Abstract
Physics-based human-object interaction has achieved robust single-agent manipulation skills, yet extending them to multi-agent cooperative tasks remains challenging. Existing approaches typically adapt interaction policies through task-specific fine-tuning, which entangles low-level contact-rich execution with high-level coordination and limits reuse across object geometries, interaction types, and team sizes. We propose a hierarchical framework that converts a single-agent HOI policy into a reusable Object-oriented Motion Skill. Specifically, we reinterpret teacher rollouts as object-oriented action supervision by extracting short-horizon object-proxy motions from executed trajectories, and distill task-specific teachers into a low-level skill operating in an Object-oriented Action Space. For downstream tasks, the distilled skill is frozen as a reusable executor, while a high-level policy coordinates multiple agents by generating region-wise object-oriented actions conditioned on the shared object, task goal, agent states, and local manipulation regions. This formulation shifts multi-agent HOI learning from direct contact-rich full-body control to compact object-level proxy-motion coordination. Experiments on diverse HOI tasks show that the distilled Object-oriented Motion Skill supports robust proxy-motion execution and enables composable policy learning across different interaction types, object geometries, and team sizes.