Skill assembly and imitation enable robots to perform complex mobile tasks
SAKI: Skill Assembly and Kinematic Imitation from Human Videos for Long-Horizon Mobile Manipulation
Robotics
Summary
Robots often struggle with long tasks that involve moving around and manipulating objects. This paper presents a system called SAKI that learns skills by watching human videos, then combines and adapts these skills to complete longer, multi-step tasks. The system keeps track of objects and robot positions to smoothly switch between skills while adjusting to changes in the environment. Tests show robots can tidy and wipe using skills originally demonstrated independently.
What this means in practice
- •For robotics engineers: Build mobile robots that reuse and combine demonstrated manipulation skills for tasks like tidying or cleaning in varied environments.
- •For logistics operators: Deploy mobile manipulators that adapt multi-step interactions from videos to efficiently handle tasks involving object organization across changing layouts.
Authors
Yijie Lu, James Zhao, Weiming Zhi
Abstract
Learning from human videos offers a promising route to acquiring diverse manipulation skills. Extending this capability beyond tabletop settings to long-horizon mobile manipulation requires adapting and composing demonstrated interactions across changing scenes and robot configurations. We present Skill Assembly and Kinematic Imitation (SAKI), a framework connecting human-video skill acquisition, cross-demonstration assembly and closed-loop whole-body execution. SAKI prepares reusable object-centric skills that preserve task-critical interactions while allowing transfer paths to adapt. Given a goal and supplied task dependencies, it selects and orders skills, binds their object roles to the current scene, and carries scene estimates and robot configuration between successive skills. Whole-body kinematic imitation generates coordinated base, arm and gripper motion. During execution, persistent object estimates maintain task references across viewpoint changes, while visual feedback updates remaining trajectories. Real-robot experiments demonstrate skill reuse across layouts and the composition of independently demonstrated interactions into continuous mobile tasks, including tidying and wiping. Ablation results show that task-conditioned reference preparation substantially improves long-horizon task completion with whole-body optimisation and visual feedback held fixed. Check https://aus.bot/research/saki/ for video demos!