Video model guides robot hands for complex long tasks
Dream What I Can: Sparse Visual Planning for Long-Horizon Robot Manipulation
Robotics
Summary
Robots need to plan steps to do long tasks like folding boxes, but predicting every move can be hard and may fail. The authors created DreamCan, a system that breaks down tasks into fewer important visual goals using videos, making it easier for robots to reach those points and complete the task. They trained DreamCan using only videos without robot actions, and it works better than other methods on both simulated and real-world robot tasks. This approach helps robots handle long, complicated jobs with fewer mistakes.
What this means in practice
- •For industrial robotics teams: Improve robot automation for long, complex assembly tasks by generating sparse visual goals that guide precise manipulation without extensive manual programming.$Commercial implications: Enables deployment of adaptable robotic systems for manufacturing that can handle diverse products and tasks efficiently with learned visual planning.
- •For robotics software engineers: Build robust robot control systems that use offline videos to create hierarchical plans, improving success rates on manipulation tasks with fewer demonstrations.
Authors
Chongkai Gao, Wenlong Huang, Haonan Chen, Lin Shao, Yilun Du, Jiajun Wu, Ruohan Zhang, Li Fei-Fei
Abstract
Video models provide rich priors for generalizable robotic manipulation, but plausible visual futures are not always executable. We introduce DreamCan, a hierarchical policy that grounds video models through sparse visual planning: a high-level planner adapted from a pre-trained video model and a low-level visuomotor policy. Rather than predicting dense trajectories, the high-level planner generates sparse visual subgoals that balance reachability and task utility. From offline, action-free robot videos, we estimate these quantities and use dynamic programming to select subgoal sequences without manual annotations. The resulting subgoals are more reliably attained than uniformly spaced alternatives, and the full system outperforms language, dense-video, and uniform-subgoal planners, reaching high planning success rates on Language-Table tasks and 91.2% success on LIBERO-Long with a 0.1B-parameter low-level policy. Additional suboptimal videos and test-time compute further improve success. The system also performs challenging real-world bimanual tasks, including box folding and packing with unseen object combinations. More results are available at https://dreamcan.github.io/.