MATS: A novel multi-modality multi-task learning framework for 3D perception in autonomous driving
2026-07-27 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors designed a new way to combine data from cameras and LiDAR sensors to better understand 3D scenes for self-driving cars. Instead of mixing all sensor information into one shared map, their method creates multiple specialized maps that suit different tasks, like detecting objects or understanding the environment. They also use a smart system that picks the best map for each task automatically. Tests show their approach works better than previous ones on a big self-driving car dataset.
3D perceptionmulti-modality fusionbird's-eye-view (BEV)multi-task learningMixture-of-Experts (MoE)LiDARcamera sensorsnuScenes datasetautonomous driving
Authors
Junchen Huo, Wanming Hao, Song Wang, Enqing Chen, Shouyi Yang, Guanghui Wang
Abstract
Multi-modality data from different sensors provides rich complementary information for 3D perception, becoming an essential component in reliable autonomous driving systems. Current research typically designs intricate and complex fusion strategies to integrate information from multimodal data on a unified bird's-eye-view (BEV) feature map for the joint learning of multiple perception tasks. However, such a single feature map hardly carries sufficient information to simultaneously meet the requirements of various perception tasks, leading to a very limited perception performance. To mitigate this limitation, this paper proposes MATS, a novel multi-modality multi-task learning approach with modality-adaptive BEV fusion and task-specific Mixture-of-Experts (MoE) for 3D perception. Specifically, a simple modality-adaptive BEV fusion module is designed to adaptively recalibrate the BEV features by modeling the global cross-modality dependencies, generating diverse BEV feature maps for various perception tasks. For joint multi-task learning, this paper proposes a task-specific MoE module to decouple the tasks and enable the network to automatically choose the appropriate BEV feature candidates for each specific task. To validate the effectiveness of the proposed approach, we conduct extensive experiments on the large-scale benchmark nuScenes. With the camera- and LiDAR-modality input data, the proposed approach outperforms the state-of-the-art (SOTA) by a significant margin. Furthermore, the experimental results on the single tasks show that the proposed approach significantly outperforms the baselines. The code and trained models will be available upon publication.