Motion guided regularization improves robot control in varied scenes
Motion-guided Contrastive Regularization For Latent World Models
Computer Vision and Pattern Recognition
Summary
Some AI models learn to predict how things move and change when actions happen, but they can get confused by backgrounds that don't matter. The authors found that static but varied backgrounds can mislead these models into ignoring important changes for controlling objects. They created a new method that uses motion cues to teach the model to focus on how things actually move during an episode. This method helps the model better understand the effects of actions, leading to improved success in planning tasks involving manipulating objects, even when backgrounds change randomly.
What this means in practice
- •For robotics engineers: Improve robot planning and control in environments with visually complex or changing backgrounds using motion-aware latent models.
- •For autonomous vehicle developers: Enhance vehicle perception models to better distinguish relevant dynamic changes from static background variations during navigation tasks.
Authors
Yiwen Chen, Qiqi Hou, Jisoo Jeong, Yinhao Zhu, Huaizu Jiang, Amir Said
Abstract
JEPA-style world models learn action-conditioned latent dynamics without reconstructing observations, but distributional regularization alone does not ensure that their representations preserve information needed for control. We identify static collapse, a failure mode in which the encoder preserves episode-specific static content while suppressing manipulation-relevant state changes. Because static content is temporally predictable yet diverse across episodes, this failure can persist despite regularizers such as SIGReg. A controlled study on OGBench-Cube shows that introducing randomized backgrounds substantially degrades planning performance while leaving the underlying manipulation task unchanged. To address this problem, we propose motion-guided contrastive regularization. Our approach compares predicted latent transitions with transitions to candidate observations from the same episode, reducing the usefulness of static background identity for distinguishing action outcomes. Optical-flow similarities provide soft contrastive targets, allowing candidates with similar motion to receive similar supervision. Combined with latent prediction and SIGReg, our objective encourages representations to preserve within-episode state distinctions and similarities between related transitions. Our method improves planning success over LeWorldModel (LeWM) from 52% to 78% on OGBench-Cube with randomized backgrounds and from 14% to 54% on LIBERO-Object, while maintaining comparable performance on four other visual-control benchmarks.