Motion-Saliency Complementary Masked Modeling for Point Cloud Video Understanding
2026-08-31 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors introduce MoSaiC, a new method for teaching computers to understand 3D videos made of point clouds without needing labeled data. Their system focuses on important moving parts by gradually masking and predicting motion-related details. They also teach the model to recognize local rotations and maintain consistency from different views to better capture how objects move and look. Tests show their approach works well for tasks like recognizing actions and segmenting scenes. Overall, the authors present a way to improve 3D video understanding by combining motion and appearance learning.
point cloudself-supervised learningmasked modelingmotion saliencyLie algebra so(3)rigid rotationcross-view consistencyaction recognitionsemantic segmentationtemporal action segmentation
Authors
Wei Wang, Yiding Sun, Yuyan Wang, Zhuoyue Zhang, Zhengqiao Li, Dongfu Yin, Chen Li
Abstract
Point cloud video representation learning is crucial for 3D dynamic scene understanding. In this paper, we propose MoSaiC, a novel Motion-Saliency Complementary masked modeling framework for self-supervised point cloud video representation learning. MoSaiC couples three components: Curriculum Motion-Saliency Masking (CMSM), which guides the masking process toward motion-salient tokens under a curriculum schedule; Normal-Flow Motion (NFM) modeling, which supervises the local rigid rotation of each token in the Lie algebra so(3) as an explicit geometric motion target; and Cross-view Token Consistency Prediction (CTCP), which enforces consistency between two complementary masked views at the token level. Together, these components allow MoSaiC to effectively capture both appearance and motion dynamics. Extensive experiments on multiple downstream tasks, including action recognition, temporal action segmentation, and point-level semantic segmentation, demonstrate the effectiveness of our approach.