Frozen 3D models reveal motion cues for video segmentation tasks
ProMo3D: Probing Motion Cues in Frozen 3D Foundation Models
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Understanding how things move in videos is usually tricky and requires lots of training on movement data. The authors show that 3D models trained on still scenes actually hold hidden clues about motion that can be uncovered without retraining. By using a lightweight method to read these clues, their approach generates motion maps with little data and no extra tuning. This technique works well on several video datasets, offering a simpler and efficient way to detect motion from static 3D models.
What this means in practice
- •For video processing teams: Produce motion segmentation masks in videos using pretrained 3D models without additional motion-specific training.
- •For augmented reality developers: Use frozen 3D model features to extract motion information for improving real-time scene understanding with minimal compute.
Authors
Xiaoang Zhang, Mert Kiray, Yordanka Velikova, Manoj Biswanath, Benjamin Busam
Abstract
Feed-forward 3D foundation models have recently reshaped multi-view reconstruction, yet dynamic 4D understanding still often relies on large-scale motion supervision or hand-crafted, architecture-specific heuristics. We ask whether frozen 3D foundation model tokens already encode motion-relevant information that can be exposed by lightweight readout functions, rather than re-learning motion-specific features with high-capacity decoders. Through probing on three representative backbones, we find that static--dynamic separation is decodable from frozen token representations. Our rank-restricted and spectral analyses on $π^3$X reveal that the recovered motion cues are concentrated in a low-dimensional subspace. Building on this, we propose Probing-based Motion Distillation (PMD), which converts pairwise motion-state compatibility scores against a fixed anchor token into dense motion masks. Trained on only 160 YouTube-VOS videos using a single 16 GB GPU, PMD transfers zero-shot to DAVIS, SegTrackv2, and FBMS-59 without target-benchmark fine-tuning. With multi-view pretrained geometric backbones, PMD outperforms training-free heuristics and approaches substantially heavier supervised and self-supervised methods, suggesting that motion decoding from frozen 3D foundation models can be lightweight and data-efficient.