EgoGeneralist unifies egocentric video tasks with 3D action modeling
EgoGeneralist: Unifying Egocentric Action-Video Tasks through Video-Native Modeling
Computer Vision and Pattern Recognition
Summary
Making sense of videos showing people’s actions from their own viewpoint is hard because computers usually don’t understand how bodies move in real 3D space. The authors created EgoGeneralist, a system that combines learning from videos with understanding 3D human motion to better generate, estimate, and relate actions seen from a first-person view. This method uses a special video-based approach to link visual motions and 3D movements, making it work well even on new types of scenes it wasn’t trained on. As a result, EgoGeneralist can handle many tasks involving egocentric action videos using one single model.
What this means in practice
- •For augmented reality developers: Create interactive AR experiences that adapt to a user’s real-time body movements captured from a first-person camera.
- •For robotics engineers: Improve robots’ understanding of human actions through first-person video for better human-robot collaboration.
Authors
Yanlin Jin, Bowen Yang, Cheng Zhang, Yifan Yang, Zhe Li, Jianfei Cai, Kai Zhu
Abstract
Modern video models possess strong visual priors, yet do not natively represent human action in metric 3D space. We present EgoGeneralist, which adapts pretrained video generators to unify action-conditioned egocentric video generation, pose estimation, and joint action-video generation. Our role-adaptive action renderer provides video-native representations for multidirectional action-video learning, making metric motion usable as both a spatial control signal and a recoverable generation target. Cross-stream action-video coupling establishes close correspondence between metric motion and visual dynamics through lightweight adaptation while preserving the pretrained visual prior. Together, these designs establish a generalist framework spanning five action-video tasks. Despite adaptation only on controlled laboratory recordings, EgoGeneralist generalizes to diverse unseen scenes and interactions with effective action control and visual quality, bridging pretrained visual knowledge and metric 3D human motion.