Deep Multimodal Fusion Detection through Spatial Mask and Channel Fusion
2026-08-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMultimedia
AI summaryⓘ
The authors developed a new method to improve object detection by combining information from different types of data (modalities). They use an attention system that mixes features from each type during training to help the model learn more general patterns instead of relying on just one source. They also introduced a way to selectively pick and combine important features from each type. Their tests show this approach works well compared to other methods.
multimodal fusionobject detectionattention mechanismchannel spatial attentionsemantic mask exchangefeature fusiondual-backbone architecturelearnable channel competitioncross-modality
Authors
Guandi Wang, Ming Li, Yunsen Xing, Junle Liu
Abstract
Deep multimodal fusion for object detection has demonstrated good performance through mining modal characteristics. However, existing feature-level fusion methods mainly weigh between two modalities and unify them in a unified representation space. This can lead to overfitting or over-specialization of the statistical properties of a single modality within a dual-backbone architecture. This paper proposes an Attention-Driven Complementarity Resampling framework for robust improvement of cross-modality object detection. Based on a shared channel spatial attention mechanism, we first introduce the semantic mask exchange to actively mix the boundaries of the modalities during the training phase, forcing the backbone network to learn generalized features without relying on fixed modal labels. Then we propose a learnable channel competition to sample and aggregate features in a channel-wise and learnable way. Our experiments on multiple datasets demonstrate that the proposed method is effective and yields competitive results among existing state-of-the-art approaches. The source code is provided in the supplementary material.