Does Latent Planning Survive Point Clouds? Action-Conditioned JEPA World Models for Geometric Observations
Machine LearningArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
The authors study how a type of AI called JEPA, originally used with images, can work with 3D point clouds—collections of points representing shapes in space—to plan actions. They test three JEPA designs adapted to point clouds and find that these models can still plan well, even when parts of the scene move or data is noisy. They show that the models pay attention to moving points and can use 3D target positions directly, making it easier to set goals without extra images. Their findings suggest JEPA methods are effective beyond flat images and handle 3D geometry robustly.
JEPAlatent-space planningpoint clouds3D geometryself-supervisionfrozen encoderdistribution prioraction-sensitive modellatent predictiongoal latent
Authors
Fabio F. Oberweger, Michael Schwingshackl
Abstract
JEPA world models make latent-space planning a practical route to control, but they are built almost exclusively on images. Whether latent prediction survives geometric observations is unclear: point clouds are sparse, unordered, and self-occluded, and with 0.3-15% of scene points moving, the slow-feature optimum of latent prediction compounds with the geometric shortcut of 3D self-supervision. We lift three canonical JEPA designs to point clouds, frozen-encoder, distribution-prior, and action-sensitive, and re-sense the stable-worldmodel benchmark so that only the observation differs from the image baselines. All three plan without collapse: the distribution-prior model is statistically equivalent to its re-evaluated image counterpart on every benchmark, and the action-sensitive model attains the strongest result in our controlled comparison where the most geometry moves. Probing explains why: object positions are almost perfectly linearly decodable and attention falls on the few moving points. Planning withstands heavy dropout never seen in training, though range noise defeats the thinnest scene. Geometry finally makes a commanded 3D target a natural goal interface: we construct the goal latent from the target and the current latent, at no cost in success rate, without a goal observation.