GeoLAM learns 3D motion actions from unlabeled human videos for robots
GeoLAM: Learning Geometry-Grounded Latent Actions from Unlabeled Human Videos
Computer Vision and Pattern RecognitionRobotics
Summary
Understanding how humans move their hands during tasks can help robots learn to perform complex actions. The authors created GeoLAM, a system that learns meaningful hand motions from videos without needing labels about actions or hand poses. It uses 3D geometry to focus only on the actual movements, ignoring distractions like background changes or camera motion. After training on these videos, GeoLAM helps robots understand and replicate these actions more accurately without extra video predictions at runtime.
What this means in practice
- •For robotics engineers: Train robots to better recognize and execute hand manipulation actions using unlabeled human videos for pretraining latent action models.
- •For computer vision developers: Develop video understanding tools that isolate meaningful 3D motion from complex hand movements without needing detailed hand annotations.
Authors
Yifan Xie, Hekun Tian, Jinkun Liu, YuAn Wang, Qiao Sun, Wenbo Ding
Abstract
Human videos provide rich manipulation experience, but extracting action representations that preserve useful motion remains challenging. Visual reconstruction alone can entangle manipulation-related motion with appearance changes and camera movement. We present GeoLAM, a framework for learning geometry-grounded latent actions from action-free human videos. GeoLAM combines future-frame reconstruction through a frozen geometric feature hierarchy with motion supervision from a training-only 4D geometry teacher. The geometric representation provides a structural prior, while the teacher's predictions yield spatially pooled targets capturing 3D displacement, residual image-plane motion, and surface-orientation changes. Visibility and confidence weighting reduces the contribution of unreliable estimates, encouraging continuous latent actions to retain geometric motion without explicit hand-pose or hand-trajectory annotations. After video pretraining without action labels, the learned representation provides transition targets for a world-action model trained on action-labeled robot demonstrations. The model jointly denoises latent actions and executable action chunks, with future-video prediction used only as an auxiliary training task. Deployment therefore requires neither the geometry teacher nor future-video generation. Evaluations on a latent-action benchmark and robotic manipulation tasks demonstrate the strong performance of GeoLAM.