Hierarchical keyframe system improves humanoid robot task control
KINO: A Keyframe Interface for VLM Planning and Whole-Body Control in Humanoid Loco-Manipulation
Robotics
Summary
Humanoid robots need to understand instructions and move their whole bodies to pick up and place objects. The authors developed a system that breaks down tasks into important body poses called keyframes. A vision-language AI picks these poses based on what it sees and what the robot needs to do. Then, the robot's control system moves its joints to reach those poses, improving task success. This method was tested in simulations and on a real robot performing one- and two-handed tasks.
What this means in practice
- •For robotics engineers: Enable robots to interpret language instructions and coordinate whole-body movements for complex object handling tasks.
- •For automation system integrators: Combine sensor inputs with motion keyframes to improve robotic manipulation in dynamic environments like warehouses.
Authors
Sitong Chen, Fatemeh Zargarbashi, Jin Cheng, Tianxu An, Stelian Coros
Abstract
Humanoid loco-manipulation requires robots to interpret task instructions and scene semantics while executing coordinated whole-body motions. We propose a hierarchical framework that uses motion keyframes as an intermediate representation between Vision-Language Model (VLM) planning and Reinforcement Learning (RL) control. Each keyframe specifies a target whole-body robot pose and, when applicable, an object pose. Given a language instruction, scene observations, and execution feedback, the VLM selects successive task-relevant keyframes from a predefined library. The selected keyframes are retargeted to the current scene to account for object poses and dimensions. A keyframe-conditioned whole-body policy then generates joint-level actions to reach these goals. We introduce a saliency-based keyframe sampling strategy for low-level policy training that improves end-to-end task success rate from 44% to 92% when using sparse VLM keyframes. We evaluate our framework on object pickup, transport, and placement tasks in simulation and on a Unitree G1 humanoid. The system successfully performs both one- and two-handed manipulation and generalises to placement locations beyond the training reference data.