Unified action conditioning improves diverse world model predictions
UniAction: Unified Action Conditioning for World Models
Computer Vision and Pattern Recognition
Summary
World models predict how things will change in response to actions, but actions come in many types, making it hard to train one model for all of them. The authors propose UniAction, which breaks down any action into simple parts with defined meanings and positions in time and space, allowing a single model to handle all kinds of actions together. This approach was tested with different action types like forces and robot movements, outperforming models specialized for just one kind. Their work shows combining diverse action data in one model can actually improve how well actions are understood and predicted.
What this means in practice
- •For robotics engineers: Train a joint world model that accurately predicts outcomes across varied robot actions using unified action conditioning.
- •For augmented reality developers: Improve action prediction in AR scenarios combining camera, hand, and other controls using a shared world model.
Authors
Bowen Xue, Zizhang Li, Hong-Xing Yu, Jiajun Wu
Abstract
Actions that condition world models are highly heterogeneous, and thus existing action-conditioned world models are typically built for individual action types, fragmenting already limited action-annotated data across separate models. We introduce UniAction, a unified action-conditioning framework that represents any action as a sparse set of action elements, each with a discrete semantic identity and a continuous spatiotemporal grounding coordinate. By mapping these coordinates directly into the pretrained video model's native positional space, UniAction allows heterogeneous actions to condition a shared world model within the same spatiotemporal structure. We further introduce cardinality-calibrated attention to address the bias from differing action representation granularities. We evaluate UniAction on four representative action forms: force, camera control, hand pose, and robot pose. UniAction outperforms conditioning alternatives in action following, and a single jointly trained model surpasses the action-specific experts, showing that heterogeneous action data benefit from unified model and joint training.