Occlusion treated as masking for faster amodal video segmentation
OccJEPA: Occlusion as Masking for Amodal Video Segmentation
Computer Vision and Pattern RecognitionMachine Learning
Summary
Seeing the full shape of objects in videos, even parts hidden behind others, is a tough problem. The authors treat hidden parts as masked areas and use a special predictor trained to guess these missing pieces from visible video parts. Their method is much faster and more accurate than previous ones, working without extra depth data and producing consistent results. This technique helps computers better understand full object shapes in complex video scenes.
What this means in practice
- •For video analytics teams: Generate complete object masks in videos faster and more reliably, even when parts are hidden, improving scene understanding for surveillance or content analysis.
- •For augmented reality developers: Create realistic occlusion-aware object masks to improve overlay accuracy when virtual objects interact with real ones in video streams.
Authors
Genpei Zhang
Abstract
Amodal video segmentation asks for the complete mask of an object in every frame, including the parts hidden behind other objects. The best current methods generate these masks with fine-tuned video diffusion models, which need many denoising steps and an extra depth network, and give a different answer for every sample. Self-supervised video world models of the JEPA family are trained on a closely related task: predicting the representation of masked space-time regions from the visible ones. We therefore treat occlusion as masking. The tokens that the object's visible mask does not cover are dropped, the pre-trained V-JEPA 2.1 predictor fills them in, and a light head decodes the observed and predicted features into amodal masks. The resulting model, OccJEPA (Occlusion-as-masking JEPA), is deterministic and 33.5x faster than the diffusion-based state of the art, and more accurate on SAIL-VOS, on TAO-Amodal, including a held-out test split (+4.00 AP50) and visible masks tracked by SAM2, and on Kubric. Controlled experiments trace the gain to the predictor's pre-trained weights rather than to its capacity, for world models from 87M to 1.85B parameters. Code is available at https://github.com/zgp110/occjepa.