Multimodal fusion boosts chest X-ray diagnosis accuracy
Guided Co-Attention with Adaptive Gating for Multimodal Medical Data Fusion
Computer Vision and Pattern RecognitionMachine Learning
Summary
Combining information from different sources like images and text can help make better medical decisions. This paper shows a way to merge chest X-rays and their reports so that both types of information help each other smartly. The method treats image and text differently and decides how much each should influence the result for every case. This approach led to very accurate detection of normal versus abnormal findings in chest X-rays. The authors found that considering both directions of interaction and adjusting the mix per sample improves performance.
What this means in practice
- •For radiology it teams: Improve diagnostic support systems by integrating chest X-rays with radiology reports using adaptive fusion to boost classification accuracy.
- •For medical imaging software developers: Develop smarter medical image analysis tools that dynamically balance text and image information to better detect abnormalities.
Authors
Muntaqim Ahmed Raju, Priyanka Siddappa, Md Shifat Haider Al Amin, Ruizhe Ma, Md Ripon Ahammed
Abstract
Multimodal fusion requires not only exchanging information across modalities but also determining which representations should participate in the interaction and how complementary cross-modal views should be combined. We introduce Guided Co-Attention with Adaptive Gating, a fusion framework that integrates token-wise guidance, bidirectional cross-attention, and sample-specific fusion. The proposed method constructs two directional representations: an image-conditioned text representation, obtained by allowing textual tokens to query visual features, and a text-conditioned image representation formed in the reverse direction. Token-wise guidance modulates the representations before cross-attention, while an adaptive gate learns the relative contribution of the two resulting directional representations for each sample. We evaluate the approach using paired chest radiographs and radiology reports as an image-text test bed for binary (normal vs. abnormal) classification. The two cross-attention directions exhibit distinct performance characteristics, demonstrating that reversing the query and source modalities does not produce interchangeable representations. Ablation experiments further show that bidirectional interaction, adaptive gating, and token guidance provide complementary benefits. The proposed model achieves 96.05% accuracy, 96.10% F1-score, 98.37% AUC, 95.60% Sens@95Spec, and 96.51% Spec@95Sens, yielding the highest mean performance across all evaluated metrics. These findings support explicitly modeling interaction direction, token relevance, and sample-dependent integration when constructing multimodal representations.