Whole body control combines dexterity and scene interaction skills
Co${}^{2}$Skill: Whole-Body Control via Skill Composition for Long-Horizon Human-Environment Interaction
RoboticsGraphics
Summary
Controlling a whole robot body to manipulate objects while interacting with its surroundings is very hard. The authors created Co2Skill, a system that learns to do both things together by combining pre-learned motions and adapting based on goals. It can perform tasks like sitting, climbing, and handling objects in a sequence, keeping control smooth across different actions. Their approach helps robots move and use their hands in a coordinated way to achieve complex goals in everyday environments.
What this means in practice
- •For robotics engineers: Train robots to perform coordinated whole-body and hand-object tasks like climbing and manipulating objects in complex indoor settings.
- •For automation integrators: Implement unified control policies that enable robots to handle sequences of tasks involving movement and object manipulation seamlessly.
Authors
Jeonghwan Kim, Hyeonwoo Kim, Hanbyul Joo
Abstract
Achieving human-level dexterity in complex, unstructured environments requires the seamless integration of whole-body scene interaction and dexterous object manipulation skills. While existing physics-based controllers generate physically plausible behaviors in each domain, they largely address these two capabilities independently. In this paper, we present Co${}^{2}$Skill that integrates scene interaction and dexterous manipulation through a unified policy formulation. Built on a pretrained motion prior, the policy uses task and phase dependent observation masks to select information relevant to the current interaction goals. We introduce a goal-conditioned loco-manipulation curriculum that combines partial reference guidance for precision with exploration from varied initial states while allowing goal-directed execution beyond the demonstrated trajectories. We further introduce a cross-task curriculum that jointly trains individual skills and selected task sequences, preserving physical states across task boundaries and maintaining grasps during subsequent scene interactions. Together, these support sequential task execution and simultaneous scene interaction with object manipulation. We evaluate sitting, standing, climbing, stair traversal, and goal-directed manipulation, together with sequential execution and with random different conditions. Additionally, we demonstrate skill compositions in indoor environments, illustrating their integration within the same control formulation.