Large language model improves medical image question answering accuracy
A visual large language foundational model for medical image recognition using clinician-oriented social media
Artificial Intelligence
Summary
Answering questions about medical images is hard because there are not enough datasets showing how doctors think and connect images with text. The authors collected over a million pairs of medical images and doctor comments from social media, creating a large, detailed dataset called ThoughtMed-1M. Using this dataset, they trained a new model, FOLTMed, that performs better than previous methods on many medical image question tasks. This shows a promising way to teach computers to understand medical images more like clinicians do.
What this means in practice
- •For medical imaging software teams: Improve diagnostic support tools by integrating FOLTMed for more accurate answers to medical image queries using clinically verified data.$Commercial implications: Enables advanced clinical decision support software that provides reliable image-based answers to doctors, enhancing diagnostic accuracy and workflow.
- •For clinical informatics teams: Use ThoughtMed-1M dataset to train or benchmark AI models for medical imaging tasks with clinically aligned image-text data.
Authors
Lingxuan Hou, Yuhua Xie, Yue Hu, Yan Zhuang, Junqi Li, Chengzhi Xia, Binh Phu Nguyen, Abubakar Siddique, Minh Nguyen, Yao Hou, Yanju Bao, Kexin Liu, Ke Chen, Jianjun Sun, Zeqi Li, Trung Nguyen, Jiangli Lin
Abstract
Large language models (LLMs) have demonstrated strong capabilities across diverse domains, showing considerable potential in medicine. However, their application in medical settings remains limited by the scarcity of visual question answering (VQA) datasets that capture clinical reasoning and explicit image-text alignment. Here, we leverage de-identified medical images and expert commentaries shared on clinician-oriented social media. By combining an advanced LLM with clinician-in-the-loop verification, we established a rigorous pipeline to construct ThoughtMed-1M, a long-form medical VQA dataset containing over one million VQA pairs and designed to capture structured clinical logic and medical image-text alignment. To demonstrate its utility, we developed a FOundational LLM Trained on ThoughtMed-1M (FOLTMed). FOLTMed achieved state-of-the-art performance across 42 medical VQA benchmark datasets, with a macro accuracy of 85.4%, and generated more clinically coherent responses on the ThoughtMed-1M test set. It outperformed state-of-the-art models by 3--5% across factuality and similarity metrics, highlighting a scalable paradigm for advancing research on clinically grounded multimodal LLMs.