PRISM: Predictive Recomposition via Semantic Latent Decomposition for View-invariant Video Representation Learning
2026-08-31 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors address the challenge of recognizing actions in videos taken from different viewpoints, like first-person (egocentric) and third-person (exocentric), where appearances can change a lot. They find that previous methods mix together features that are specific to the view and those that are common across views, which reduces accuracy. Their method, PRISM, splits video features into view-invariant and view-variant parts and then recombines them under language guidance to keep these parts separate. This approach helps the model learn better and improves performance on multiple benchmarks, even without prior training on those specific domains.
cross-view videoview-invariant featuresview-variant featuresvideo representation learningegocentric videoexocentric videofeature disentanglementzero-shot learninglanguage supervisionvideo embedding
Authors
Youngchae Chee, Hosu Lee, Sungjune Park, Junho Kim, Yong Man Ro
Abstract
Cross-view video representation learning aims to capture viewpoint-invariant action semantics despite substantial appearance changes across egocentric and exocentric videos. However, existing methods encode each video as a unified embedding, where view-invariant and view-variant semantics inevitably entangle under co-occurrences - a failure mode we show persists even in cross-view methods explicitly trained for view-invariance. Our key insight is that a view-invariant feature is truly disentangled when it can be sufficiently recomposed with an arbitrary view-variant feature while preserving their independent semantics. Building on this, we propose PRISM, that decomposes video into view-invariant and view-variant latents and recompose them under language supervision encouraging clean decomposition of the two streams. PRISM achieves state-of-the-art results on EgoExo4D, EgoExoLearn, AE2, even surpassing in-domain models under zero-shot setting. Code is available at https://github.com/litcoderr/prism.