Multimodal learning improves CT prediction of lung cancer invasiveness
Multimodal Prompt Learning with Dual-Granularity Contrastive Alignment for Preoperative Prediction of Lung Adenocarcinoma Invasiveness Using 3D CT
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
It is hard for doctors to tell whether certain lung cancers are invasive just by looking at CT scans because they can look very similar. The authors created a computer method that combines 3D CT images with written clinical information about the scans. Their system learns both the overall features of the tumor and details about nearby blood vessels to better predict if the tumor is invasive. Tests showed that this method was more accurate than just using the CT images alone. This approach could help doctors plan better treatments before surgery.
What this means in practice
- •For medical imaging developers: Create software tools that improve lung cancer invasion predictions from CT scans using combined image and radiology report data.
- •For clinical workflow designers: Design preoperative decision support systems that integrate dual-granularity radiological features without requiring text input at test time.
Authors
Qing Huang, Xiaotong Tan, Lintong Yao, Wufeng Xue, Ruobing Huang, Dong Ni, Hao Wu, Tian Xiao, Jun Cheng
Abstract
Accurate preoperative prediction of lung adenocarcinoma invasiveness is important for surgical planning and individualized treatment. However, overlapping CT appearances make it challenging to distinguish invasive adenocarcinoma (IAD) from Pre-IAD lesions. We propose a multimodal prompt-learning framework with dual-granularity image-text alignment for 3D CT-based IAD classification. A 3D ResNet-CBAM encoder extracts global and spatial visual features. A frozen RadBERT backbone is adapted with separate learnable prompts for report-level and keyword-level text encoding. These prompts are initialized using semantic representations derived from real-world clinical reports. Global contrastive alignment connects whole-volume CT representations with reports generated from clinician-annotated radiological attributes to provide overall semantic supervision. A multiple instance learning-based local alignment module associates spatial CT features with vascular-focused radiological keywords, to capture local imaging characteristics potentially associated with tumor invasion. The framework jointly optimizes classification and alignment objectives, while inference requires only CT images. Experiments included 1,593 pathologically confirmed nodules. On the test set, the proposed method achieved an accuracy of 87.92%, F1-score of 87.78%, and AUC of 0.9499. Its AUC significantly exceeded that of the image-only Res-CBAM baseline with the same visual backbone (0.9297; DeLong test, p = 0.0078). Ablation analyses supported the complementary contributions of global and local alignment and the benefit of report-based prompt initialization. Qualitative visualizations showed activation within nodules and adjacent vascular structures. These findings suggest that dual-granularity radiological supervision can enhance CT-based invasiveness prediction while preserving a CT-only inference workflow.