CheXFuse improves chest X-ray classification by fusing images and reports
CheXFuse: Guided Differential Joint Attention Fusion for Robust Multimodal Chest X-Ray Classification in Data-Scarce Medical Settings
Computer Vision and Pattern RecognitionMachine Learning
Summary
When computers look at chest X-rays and their text reports together, the challenge is that sometimes the images or the reports are stronger than the other, which can confuse the computer. The authors developed CheXFuse, a method that smartly guides attention to the most useful parts of both images and text, reducing noise from weaker signals. They tested it on a dataset of chest X-rays and found it works well even when the text reports are incomplete or less informative. This shows their method can combine different kinds of information effectively for better medical diagnosis support.
What this means in practice
- •For medical imaging teams: Improve diagnostic models by robustly combining X-ray images and clinical text reports, especially when text completeness varies.
- •For healthcare ai developers: Build chest X-ray analysis tools that maintain high accuracy even with incomplete or partial patient reports.
Tested on one dataset.
Authors
Muntaqim Ahmed Raju, Priyanka Siddappa, Md Shifat Haider Al Amin, Ruizhe Ma, Md Ripon Ahammed
Abstract
A central challenge in multimodal learning is how to fuse modalities that differ substantially in predictive strength. When one modality provides a much stronger representation than another, indiscriminate cross-modal interaction can introduce weak, redundant, or unstable signals, limiting or even degrading performance relative to the strongest unimodal representation. To address this problem, we propose CheXFuse, a multimodal fusion framework based on Guided Differential Joint Attention (GuDJA). CheXFuse combines image and text token representations within a joint multimodal sequence, where learned token guidance emphasizes informative representations and differential attention reduces less useful attention responses. We evaluate CheXFuse on the Indiana University (IU) Chest X-ray dataset, using binary (normal vs. abnormal) classification as the downstream task. Experiments are conducted under both original-report and masked-report settings, with the latter serving as a controlled stress test under reduced textual informativeness. CheXFuse with GuDJA achieves the strongest mean overall performance across the evaluated baselines and ablations, reaching 95.73% accuracy, 95.79% F1-score, 96.54% recall, and 95.28% sensitivity at 95% specificity (Sens@95Spec) with original reports, and 95.58%, 95.60%, 95.60%, and 94.34%, respectively, under masked reports. These results suggest that GuDJA provides a promising fusion strategy under asymmetric or partially reduced modality informativeness.