Robust Multimodal Dynamic Object Segmentation
2026-07-20 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors tackle the problem of separating moving objects from still backgrounds in videos, which is important for rebuilding scenes from dynamic footage. They combine different types of information like 2D points, 3D shapes, and semantic labels to better identify moving parts. Their method uses a Transformer-based network that picks which features are most useful for each scene, improving accuracy especially near object edges. They also create a new technique to refine masks for multiple objects. Tests show their method works better than existing approaches for segmenting moving objects and reconstructing static scenes.
dynamic object segmentationoptical flow3D reconstructionTransformer networkfeature clusteringsemantic informationpoint tracksmask refinementstatic scene reconstruction
Authors
Zhe Xin, Hanzhi Chang, Penghui Huang, Yinian Mao, Guoquan Huang
Abstract
Dynamic object segmentation plays a critical role in many visual applications such as static scene reconstruction from dynamic videos. However, existing optical flow-based methods fail to ensure consistent static/dynamic segmentation along object boundaries, while 3D reconstruction-based approaches are highly sensitive to reconstruction errors. To address these limitations, we present a dynamic object segmentation framework that can generate both precise and complete dynamic masks by integrating multimodal cues including 2D point tracks, 3D reconstruction, and semantic information. We design a network combining Transformer architectures with feature clustering aggregation modules to perform static/dynamic classification of multimodal feature trajectories. It enables the model to adaptively determine which type of feature should dominate based on the characteristics of each scene, while also mitigating the impact of feature degradation. Additionally, we introduce a novel point-query-based SAM post-processing method capable of handling multiple objects within a single mask. Extensive experiments demonstrate that our approach achieves state-of-the-art performance in both dynamic object segmentation and static scene reconstruction tasks.