FloMo uses 3D motion to improve robot action prediction from video
FloMo: 3D Scene Flow as a World Action Model Intermediate for Learning from Human Video
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Videos contain a lot of information, but not all of it helps robots understand how to act. The researchers found that focusing on dense 3D motion—how things move in three dimensions—is a better way for robots to learn actions from videos of humans. They created a model called FloMo that predicts 3D motion and robot actions together, which makes learning more efficient and helps robots handle new objects and tasks. The model works better when it focuses on motion rather than trying to predict full video frames alongside actions.
What this means in practice
- •For robotics developers: Create robot control policies that learn more efficiently by using 3D motion prediction from human video data.
- •For motion capture engineers: Improve motion representation in video-based systems by encoding 3D scene flow for better action understanding and transfer.
Authors
Jeremy Collins, Namra Patel, Ayush Agarwal, Shitij Govil, Animesh Garg
Abstract
Pretrained video generation models distill internet-scale motion priors into robot policies, but full video is a redundant signal for action prediction. We argue that dense 3D motion is the medium through which video pretraining best transfers to robot actions, and that predicting motion rather than video affords better sample-efficiency for action prediction. FloMo is a 5B video-generation model fine-tuned to predict 3D scene-flow and action tokens. Rendering scene flow as RGB and encoding it into the same latent space as video enables a pretrained video generation backbone to efficiently adapt to predict 3D scene flow. Co-trained on action-free human video and action-labeled robot teleoperation data, our model benefits from human-video pretraining and generalizes to unseen objects, motion primitives, and tasks. Replacing motion prediction with a video prediction objective degrades out-of-distribution generalization, while adding a video prediction target alongside motion lowers policy success rather than improving it. When motion, video, and action are predicted jointly, action tokens attend far more to motion than to video, suggesting that predicting video wastes model capacity. Together, these results indicate that motion effectively mediates the transfer from video pretraining to action. More information can be found at flomo-wam.github.io.