Inside-jepa grounds relations in latent world models for action prediction
Inside-JEPA: From Prediction to Relational Grounding in Latent World Models
Machine LearningComputer Vision and Pattern RecognitionRobotics
Summary
Predictive models often learn hidden world states that are hard to interpret. The authors introduce Inside-JEPA, a method that embeds relationships between objects directly into these hidden states and uses a training signal based on predicting actions to help the model understand these relationships without needing labels. They show this improves the model’s ability to recognize and predict relations and actions in video and robot tasks. This approach helps models better make sense of how things interact in their environment.
What this means in practice
- •For robotics engineers: Use grounded relational channels in latent space to improve robot action recognition and planning in new or unseen tasks.
- •For video analytics developers: Implement relationally grounded latent representations for better understanding of object interactions in complex video datasets without needing relation labels.
Authors
Hossein Resani, Javen Qinfeng Shi
Abstract
Joint-embedding predictive architectures (JEPAs) learn world models in latent space, enabling efficient planning but leaving their states opaque. We introduce Inside-JEPA, which places relational channels for ordered entity pairs inside the JEPA latent state and shows that an inverse-dynamics loss can ground these channels without relation labels when its head reads them, either alone or as part of the full state. On Something-Something V2, latent prediction alone performs no better than a random relation head, while inverse-dynamics supervision that recovers actions from channel changes improves relational AUROC by .065-.103 across ten settings with annotated boxes, by .081 on RH20T robot episodes with held-out tasks, and by .050-.061 across five relations using off-the-shelf detection and tracking. An inverse head on entity latents decodes actions more accurately but barely improves relational grounding; shuffled labels, two label-free objectives, and action labels on only 5% of the pairs also fail to ground the channels. Given an action, predicted relational changes approach the quality of observed future relations (AUROC .599 vs .621). For one-step relational goals, a supervised readout of the 16 channels ranks the correct action among the top five of 65 in 43% of cases, compared with 24% for prediction-only latents and 17% for a frequency prior.