Summary
Learning good internal representations of moving scenes from videos is tricky, because simple methods tend to focus only on slow or unchanging parts, ignoring important motion details. The authors introduce a new technique called MotionJEPA that encourages the model to notice both static parts and dynamic changes without needing to reconstruct full images or use action labels. This makes the learned features richer and more balanced, allowing better understanding and planning in environments with distractions. Tests show that MotionJEPA captures useful information about object motions and how they evolve over time, helping with tasks like planning paths in tricky scenes.
What this means in practice
- •For robotics engineers: Enable robots to better plan actions in environments with distracting static backgrounds by learning balanced motion representations.
- •For computer vision developers: Improve video understanding systems by embedding dynamic scene changes in latent spaces without requiring action labels or image reconstruction.
Authors
Markus Karmann, Shile Li, Christian Internò, Bruno Andreis, David Klindt, Randall Balestriero, Jindong Gu, Philip Torr, Qi Zhang, Peng-Tao Jiang, Hao Zhang, Bo Li, Onay Urfalioglu
Abstract
Joint Embedding Predictive Architectures (JEPAs) are a promising paradigm for learning task-agnostic latent world models without visual reconstruction. However, standard JEPA training exhibits a strong inductive bias towards slow features, causing feature suppression and the collapse of latent representation. While inverse dynamics provides temporal anti-collapse, it relies on action labels and offers little incentive to embed general, unlabeled dynamics. We introduce Difference Image and Single image embedding Regularization (DISReg), a novel regularizer that builds on an inverse-dynamics-style module that predicts temporal difference image embeddings without any pixel reconstruction loss, encouraging balanced static and dynamic feature learning. DISReg consists of a static term that shapes the distribution of the image embedding and encourages slow features, and a dynamic term, which, unlike direct regularization on the embedding, imposes no constraint on the image embedding's shape or distribution and instead only incentivizes that dynamic features be present. By integrating this regularizer into a standard JEPA, we establish our new architecture, MotionJEPA. Latent probing demonstrates that MotionJEPA produces more complete representations than other methods, and our trajectory analysis shows it maintains geometrically simple latent embeddings with low curvature. We further show that MotionJEPA improves downstream planning success under static-background distractors across four environments.