Self-supervised learning improves analysis of whole-body pet ct scans
LoRe-MIM: Lost-or-Replaced Masked Image Modeling for Self-Supervised Representation Learning in Whole-Body PET/CT
Computer Vision and Pattern Recognition
Summary
Analyzing whole-body PET/CT scans is important for detecting diseases and planning treatments but requires lots of expert work to label images, which is slow and costly. The authors created a new way for computers to learn from lots of unlabeled PET/CT data by masking parts of the images and asking the model to guess what's missing using information from two kinds of images. This method helps the model understand both the details in each scan type and how the two scan types relate, making it better at tasks like finding cancer or reconstructing images from partial data.
What this means in practice
- •For medical imaging teams: Improve lesion detection and classification accuracy in whole-body PET/CT scans using self-supervised representations.
- •For radiology software developers: Build more robust PET/CT image reconstruction tools that leverage cross-modal masked imaging techniques.
Authors
Amartya Bhattacharya, Bashirul Azam Biswas, Biratal Raj Wagle, Michael S. Leapman, Matthew E. Maeder, James B. Yu, Indrani Bhattacharya
Abstract
Deep learning models for whole-body positron emission tomography/computed tomography (PET/CT) hold significant clinical potential for disease detection, treatment planning, and response assessment. However, the time and cost associated with expert annotations remain major bottlenecks in developing robust and generalizable models. Although self-supervised learning (SSL) has emerged as a promising approach to mitigating annotation scarcity, its application to whole-body PET/CT remains relatively underexplored due to the limited availability of large-scale datasets and the challenges associated with learning complementary cross-modal representations. To address these limitations, we curate a heterogeneous, multi-tracer, multi-institutional cohort comprising 6,531 whole-body PET/CT scans and propose LoRe-MIM (Lost-Or-Replaced Masked Image Modelling), a novel self-supervised framework designed to learn multimodal representations through masked image reconstruction. Specifically, LoRe-MIM introduces a masking strategy in which image regions are either replaced with zeros (Lost) or substituted with corresponding complementary imaging modality (Replaced), encouraging the model to capture both modality-specific characteristics and cross-modal relationships. We evaluate the learned representations across multiple downstream tasks, including multimodal PET/CT image reconstruction, patch-based classification into cancer vs. non-cancer, whole-body lesion segmentation, label-efficient training, and robustness assessment. Experimental results demonstrate that LoRe-MIM consistently outperforms the corresponding state-of-the-art baselines across these tasks, highlighting its potential as a scalable and generalizable self-supervised learning framework for whole-body PET/CT analysis.