Multimodal AI predicts benign breast tumors to reduce biopsies
Explainable Multimodal AI Approach to Reduce Unnecessary Breast Biopsies
Artificial Intelligence
Summary
Breast biopsies can be stressful and sometimes unnecessary if a tumor isn’t cancerous. The authors created an AI system that looks at mammograms and doctor’s notes to better predict which tumors are harmless. Their system uses special techniques to also show doctors why it made those predictions. This could help doctors avoid unneeded biopsies and lower patient worry and costs.
What this means in practice
- •For radiology departments: Use an AI tool to better identify benign breast tumors from images and reports, reducing unnecessary biopsies in high-risk patients.
- •For medical imaging software developers: Integrate explainable multimodal AI combining mammograms and text reports to improve breast diagnosis accuracy with interpretable results.$Commercial implications: Enables development of AI-powered diagnostic software that lowers biopsy rates and supports clinical decision-making.
Tested on one dataset.
Authors
Sadam Hussain, Mansoor Ali, Daly Betzabeth Avendaño-Avalos, Jose Gerardo Tamez-Peña
Abstract
Recently, there has been rise in multimodal vision-language models for computer-aided diagnosis (CAD), yet their application in breast diagnosis, particularly for predicting biopsy outcomes remains limited. This study presents a multimodal vision-language approach, incorporating Grad-CAM as an Explainable AI (XAI) technique, to predict benign tumors in patients recommended for biopsy, especially those classified as BI-RADS 4 and 5. A novel multimodal framework was devel- oped using Vision Transformer (ViT) for imaging feature extraction and DeBERTa-V3-Large for textual embeddings. The model was trained and evaluated on a comprehensive in-house dataset comprising 1,632 patients, each with four mammography views and a corresponding radiology report. The features of both modalities were extracted independently and fused by late fusion before classification. Grad-CAM was used to generate visual explanations highlighting key regions influencing model predictions. The proposed model demonstrated high sensitivity 90%, 95%, 99%, and 100% with corresponding specificities of 40%, 36%, 18%, and 11%, respectively. The best AUC achieved was 0.72 by multimodal approach on the test set, outperforming the image only (0.66) and text-only (0.68) baselines. The proposed multimodal architecture, enhanced with Grad-CAM, demonstrates a strong potential to identify benign tumors among high-risk patients, offering interpretable insights and thereby helping reduce unnecessary biopsies, patient anxiety, and healthcare costs.