Shared latent actions improve control of world models across domains
OneLA: Learning Many Worlds from Latent Actions
Computer Vision and Pattern Recognition
Summary
It can be hard to teach computers how to understand and predict actions in different settings because actions look different in various types of videos and robots. The authors introduce OneLA, a method that learns a shared way to represent actions from videos without needing labeled examples. This makes it easier for a computer model to predict what will happen next in the scene, even when the background or object looks very different. Their method improves accuracy and the quality of generated video sequences compared to previous approaches.
What this means in practice
- •For robotics developers: Enable robots to use a shared latent action space for better control across different physical platforms and environments.
- •For computer graphics engineers: Generate realistic video sequences conditioned on latent actions for enhanced simulation or animation in diverse scenes.
Authors
Ziyu Zhu, Zhaoxu Sun, Fan Jiang, Chiyu Wang, Junfeng Ni, Mu Xu
Abstract
Learning world models across diverse domains calls for a unified control space, yet ac- tion interfaces vary across domains and action annotations are often scarce. Latent ac- tions learned directly from videos offer a promising way to establish such a space without explicit action supervision. However, learning shared latent actions across domains re- mains challenging, as variations in appearance, embodiment, and dynamics can entangle domain-specific visual cues with transferable transition structure. To this end, we intro- duce OneLA, a simple framework that learns continuous latent actions through a shared orthogonal dictionary and a common predictive objective, without task-specific designs or cross-domain alignment losses during latent action pretraining. These latent actions pro- vide a shared conditioning interface for world models, with lightweight adapters mapping native control commands into the same space. Conditioned on latent actions and the cur- rent visual state, the world model predicts context-dependent transitions for controllable rollout. OneLA improves action readout across domains and reduces action cycle-transfer rotation error by 24.8% over the strongest evaluated baseline. A joint-training comparison further shows a 20.5% improvement in Human Preference Score v3 (HPSv3) for real- scene generation. These results suggest a path toward general-purpose world models that learn from diverse videos and support flexible control through latent action space.