Gnomon4D improves robot action prediction using 4D motion understanding
Gnomon4D: Learning Where and When to Focus for Geometric Motion-Grounded World-Action Modeling
Computer Vision and Pattern Recognition
Summary
Predicting how robots should act based on video is tricky because changes in what we see mix object movement and appearance changes, and depth maps miss movement over time. The authors developed Gnomon4D, a model that creates future video, robot actions, and a special 4D motion map showing how both the robot and objects move over time. This helps the robot focus on important interacting parts and better decide what to do next. Their method performs better than previous ones, especially when tested on new situations.
What this means in practice
- •For robotics engineers: Enhance robot manipulation by using Gnomon4D to predict future motions and actions more accurately for physical tasks.
- •For computer vision developers: Use Gnomon4D’s 4D motion fields to improve video-aligned motion understanding in AI systems requiring spatial-temporal object interaction analysis.
Authors
Lingwei Dang, Yan Zhou, Juntong Li, Zhenhao Zhang, Yiteng Chen, Xiaoqiang Liu, Liang An, Hongwen Zhang, Wei Min, Guoxin Zhang, Pengfei Wan, Yebin Liu, Qingyao Wu
Abstract
World-action models (WAMs) use video-generation priors to jointly predict future observations and robot actions. Yet RGB futures entangle physical state changes with appearance, while frame-wise depth and point maps lack temporal correspondence. Existing 4D approaches either keep motion implicit in visual features or translate predicted traces into actions through a separate interface. We present Gnomon4D, a WAM that jointly generates future video, robot actions, and GeoDelta, a video-aligned 4D geometric motion field encoding the cross-frame image-plane displacement and depth change of embodiment and object points. By placing embodiment motion and object responses in a shared space, GeoDelta exposes their motion relations to the action expert at inference. Interaction-Salient Attention (InSAttn) uses these relations to focus spatial readout on task-relevant interaction regions, while Transition-Anchored Aggregation (TransAgg) aggregates evidence around changes in their relative motion, such as grasp and release. GeoDelta describes surface motion independently of embodiment-specific controls, providing a shared human--robot quasi-action for action-free pretraining on EgoManip-5000H followed by robot mid-training on Robot-30H. Gnomon4D leads on RoboTwin~2.0, VLABench, and RoboDojo, and outperforms the strongest real-robot baseline by 7.0 and 16.0 percentage points under in-distribution and out-of-distribution settings, respectively.