Selective aggregation improves 3D driving scene understanding
SelectOccFlow: Selective Spatiotemporal Aggregation for 3D Occupancy and Scene Flow Prediction
Computer Vision and Pattern RecognitionRobotics
Summary
Understanding 3D scenes for self-driving cars requires knowing what objects are where and how they move. The authors found that current methods struggle because they mix incompatible image information over time and space, causing errors. They developed SelectOccFlow, which carefully chooses what image and past data to use, focusing on compatible features and refining 3D shapes better. This approach showed improved accuracy and robustness to visual problems in standard tests.
What this means in practice
- •For autonomous vehicle engineers: Enhance 3D perception modules by improving occupancy and motion predictions using selective aggregation of image and temporal data.
- •For robotics system developers: Improve robotics navigation in dynamic environments by adopting refined 3D scene and flow estimation techniques resilient to visual noise.
Authors
Yuhang Wang, Kai Luo, Yuanfan Zheng, Kailun Yang
Abstract
Comprehensive 3D scene understanding for autonomous driving requires modeling geometry, semantics, and motion. However, camera-based occupancy and scene flow prediction are sensitive to unreliable spatial and temporal aggregation, caused by semantically incompatible image features, misaligned historical observations, and incomplete voxel structures. To address this issue, we propose SelectOccFlow, a selective spatiotemporal aggregation framework that progressively refines contextual evidence across image, temporal, and voxel domains. To obtain semantically compatible image evidence, we design Semantic-Guided Sampling (SGS) to regulate feature sampling with semantic priors. Since reliable image evidence alone cannot resolve temporal inconsistency, we then present State-Conditioned Temporal Aggregation (SCTA) to selectively retrieve historical evidence according to voxel states. To further enhance the structural completeness of voxel representations, we introduce Extent-Aware Spatial Aggregation (ESA), which exploits directional structural support to refine foreground geometry. Experiments on OpenOcc demonstrate that SelectOccFlow achieves a state-of-the-art OccScore of 44.9, improving the previous best by +4.2%. It also maintains competitive occupancy performance on Occ3D-nus and improves the mean OccScore under nuScenes-C corruptions by +11.1%, demonstrating improved robustness to visual corruptions. The source code will be made publicly available at https://github.com/muchen1021/SelectOccFlow.