Video plans improve versatile robotic hand manipulation in simulation
Grounding Generated Video Plans in Simulation Towards Versatile Dexterous Controllers
Robotics
Summary
Controlling robot hands to manipulate objects is hard because they need precise movements. The authors show how computer-generated videos of hand-object interactions can be used to teach robots by translating these videos into control plans. Their method makes it easier to create many examples and helps robots perform various grasp and object-moving tasks better than previous methods. It works both in simulation and on real robots, enabling more versatile and reliable robotic manipulation.
What this means in practice
- •For robotics engineers: Enable robots to perform diverse grasping and object manipulation tasks more reliably using video-generated motion plans grounded in simulation.
- •For automation system developers: Integrate scalable video-grounded controllers in simulation to improve manipulation policies in tasks requiring dexterous multi-object handling.
Authors
Tianyue Wu, Boyuan An, Shuqi Zhao, Heyu Guo, Wanli Xing, Yi Ma, Kaifeng Zhang, Ruihai Wu, Masayoshi Tomizuka
Abstract
Generated hand-object interaction (HOI) videos provide a controllable way to propose manipulation motions. Simulation-based HOI tracking can translate such kinematic references into feasible low-level control, but its scalability is limited by the lack of reliable reference motions. We therefore combine generated videos with simulation-based HOI grounding: during training, generated videos provide diverse motion references for learning a multi-object, multi-trajectory HOI tracker, and at deployment, the video model produces motion plans that are executed by the learned tracker. In particular, we propose a method that enables scalable reference generation by HOI reconstruction with minimal manual intervention and successfully grounds more than 1,500 generated videos in simulation, achieving success rates over 25 percentage points higher than those of baselines during simulation-based training. In real-world closed-loop experiments, it achieves diverse grasps, including functional grasps, non-prehensile manipulation, and post-grasp object-pose tracking. Videos and code are available at https://boyuan-an.github.io/GALATEA/.