Findings-aware model improves CT scan interpretation and report generation
Findings-aware Language-Image Pretraining for CT Volume Interpretation
Computer Vision and Pattern Recognition
Summary
CT scans show detailed images of the inside of the body, and doctors use written reports to describe what they find. Current AI models learn to connect images and reports but sometimes treat all different patient cases as unrelated, even when they share similar problems. The authors created a new training method that helps AI understand and link specific medical findings in both images and reports better. This method improves the AI’s ability to identify medical issues without extra training and helps make more accurate radiology reports.
What this means in practice
- •For radiology software developers: Create AI tools that better identify abnormalities in CT scans by directly linking image features with clinical findings extracted from reports.
- •For medical imaging product teams: Develop enhanced radiology report generation systems that produce more precise, findings-specific descriptions from CT volumes.$Commercial implications: Enables commercial imaging software to generate more detailed and clinically informed reports, improving diagnostic support for healthcare providers.
Authors
Takuro Shimaya, Junya Inoh, Masahiro Saiko
Abstract
Computed tomography (CT) provides diverse patient information and various clinical tasks, for which numerous deep learning models have been investigated. Recent years have seen the development of vision-language foundation models that can comprehensively extract clinical features, driven by representation learning that combines CT volumes with radiology reports. Existing approaches typically adopt Contrastive Language-Image Pre-training (CLIP)-based representation learning, where pairs from different cases are, in principle, treated as negative pairs such that the volume and report embeddings are pushed apart. In the biomedical imaging, however, treating different cases as negative pairs is not always appropriate, particularly when clinical similarity exists. In this study, we propose a supervised representation learning framework grounded in clinical knowledge from language foundation models, rather than the weak supervision used in CLIP. Specifically, we utilized a large language model to extract abnormal finding phrases from reports. We then explicitly constructed the target features that the image encoder should output, as the semantic representation of the extracted findings using a biomedical text embedding model. Our Findings-aware Language-Image Pre-training (FiLIP) outperformed weakly supervised approaches in multi-label zero-shot finding classification in both in-domain and external-domain evaluations, and also proved effective when fine-tuned for the downstream task of radiology report generation.