Deep learning combines brain scans and clinical data to diagnose alzheimer’s
A Multimodal Explainable Deep Learning Framework for Alzheimer's Disease Diagnosis using 3D Magnetic Resonance Imaging and Clinical Data
Computer Vision and Pattern Recognition
Summary
Diagnosing Alzheimer's disease early is important but challenging, and doctors use brain scans and clinical tests to help. The authors created a computer model that learns from both 3D brain images and patient information to improve diagnosis. They found that the model’s accuracy and explanations vary depending on the data type, how the information is combined, and the patient group. A key mental test consistently helped diagnosis across different groups, but the model’s attention to brain scan features changed. This highlights the need to check how explanations hold up when testing on new patients.
What this means in practice
- •For hospital data teams: Enhance Alzheimer's diagnosis by combining brain MRI scans with clinical data using deep learning models tuned for specific diagnostic tasks.
- •For medical imaging engineers: Develop diagnostic tools that provide insights into how models use brain imaging and clinical tests differently across patient groups.
Authors
Yusuf Brima, Marcellin Atemkeng, Lakshmana Rao Namamula, Antoine Vacavant
Abstract
Dementia is a major and growing global health burden, with Alzheimer's disease (AD) accounting for most cases. Timely and accurate diagnosis is central to managing this burden and increasingly depends on integrating complementary clinical and imaging information. Multimodal deep learning can combine these modalities for AD diagnosis, but how its explanations behave across modalities, fusion strategies, and cohorts remains unclear. We developed an explainable multimodal framework pairing a 3D CNN encoder for T1-weighted MRI with a feedforward network for harmonized clinical and demographic data, comparing varied model setups on three-way and pairwise diagnostic tasks using 6,479 internal records from the ADNI and 1,703 independent records from the OASIS-3. On ADNI, the tabular-only model achieved the highest three-class AUC-ROC of 0.879 and best discriminated cognitively normal (CN) versus mild cognitive impairment (MCI; 0.903), while cross-attention performed best for MCI versus AD (0.861); CN versus AD was highly discriminative overall. On OASIS-3, the vision-only model performed best (three-class AUC-ROC 0.910); CN versus MCI remained difficult, and no fusion strategy consistently outperformed single modalities across tasks and cohorts. SHAP and Integrated Gradients identified the MMSE as the dominant tabular feature in both cohorts, with global feature rankings agreeing strongly in ADNI ($ρ=0.94$) and OASIS-3 ($ρ=0.96$); CAM-based explanations, however, changed with model configuration and cohort. These findings show that multimodal performance and explanations are task, modality, fusion, and cohort-dependent: a dominant cognitive signal persisted across cohorts, but feature contributions and CAM explanations did not, underscoring the need to evaluate explainability under cohort shift rather than as a stable, intrinsic property.