Unified AI model improves prostate cancer PET CT reports and analysis
A Unified Vision-Language Model for PSMA PET/CT Report Generation, Visual Question Answering, and Lesion Segmentation
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Reading and understanding special medical images called PSMA PET/CT scans is important for treating prostate cancer. Most existing AI tools focus on just one job, like writing a report or finding cancer spots. The authors created one AI model that can write reports, answer questions about the scans, and mark cancer areas all at once. Their model worked better than other tools in making reports, answering questions, and identifying lesions accurately. This shows one AI can handle several tasks to help doctors better interpret these scans.
What this means in practice
- •For medical imaging teams: Produce accurate, interactive PSMA PET/CT reports and analysis from a single AI system for clinical prostate cancer workflows.
- •For software developers for diagnostic tools: Build integrated medical AI applications that combine image segmentation, automated report writing, and interactive question answering.$Commercial implications: Enables packaged clinical decision support products offering multi-task PET/CT image interpretation for prostate cancer centers.
Authors
Yang Xing, Jiong Wu, Savas Ozdemir, Yang Zhou, Boxiao Yu, Ying Zhang, Zheren Zhu, Chenyu You, Wei Shao, Yang Lu, Kang Wang, Tinsu Pan, Yang Yang, Kuang Gong
Abstract
Accurate PSMA PET/CT interpretation is central to prostate cancer management, yet existing PET/CT AI models typically address isolated tasks. We propose a unified PSMA PET/CT vision-language model for report generation, visual question answering, and lesion segmentation. The framework adopts an LLaVA-style architecture, comprising a PET/CT vision encoder, an MLP-Mixer projection module, a LoRA-tuned large language model, and a 3D segmentation branch. Training followed a four-stage strategy: vision encoder pretraining, projection-layer alignment, VLM fine-tuning, and final multitask tuning. Language tasks used 5,747 PSMA PET/CT datasets with paired reports, while segmentation used the PSMA subset of AutoPET. The model outperformed PET2REP and a CT-based baseline across standard report-generation metrics, improved performance across VQA question types, and achieved higher Dice and lesion-level overlap F1 than SegAnyPET and nnUNet. These results support the feasibility of a unified framework for structured, interactive, interpretable PSMA PET/CT analysis with voxel-level grounding within a single multitask model architecture.