Cross modality tumour localisation improves liver cancer surgery guidance
Reinforcing intra-operative tumour localisation with weak supervision from pre-operative appearances
Computer Vision and Pattern Recognition
Summary
Finding tumours during surgery is hard because images taken during the operation don’t show tumours clearly and change shape compared to pre-surgery scans. The authors found a way to use information learned from pre-surgery MRI scans to help locate tumours on CT images taken during surgery without needing matching pairs of images. They use a computer program to guess where tumours might be by learning weak signals from one kind of image and guide tumour detection on another. This method worked better than others tested on liver cancer cases.
What this means in practice
- •For surgical navigation teams: Enhance tumour targeting during liver surgery by improving tumour detection on intra-operative CT with models trained on pre-operative MRI.
- •For medical imaging software developers: Develop tumour localisation tools that do not require paired MR-CT training datasets, enabling more flexible cross-modality applications.$Commercial implications: Enables creation of tumour localisation software products for surgical use without need for costly paired imaging datasets.
Authors
Sidhart Sami, Ali Sharjeel, Sri Atluri, Miao Zhao, Caroline Roney, Eirini Marouli, Shaheer U. Saeed
Abstract
Limited tumour visibility on intra-operative images complicates surgical navigation, including liver hepatocellular carcinoma (HCC) targeting on computed tomography (CT) imaging. This makes labelled datasets difficult to curate for deep learning, and transferring pre-operative delineations through registration remains challenging due to intra-operative deformation and lack of paired pre- and intra-operative data. We propose to transfer tumour appearances learned from pre-operative magnetic resonance (MR) images to guide localisation on intra-operative CT through weak supervision. A classifier is trained to classify tumour presence on pre-operative images. This is applied to windows on intra-operative images, and despite the domain gap, it assigns higher tumour-presence probabilities on average to tumour-containing windows. These weak cross-domain signals guide a reinforcement learning agent to localise tumours on intra-operative images. We compare direct transfer, few-shot adaptation and synthetic transfer (trained using unpaired MR and CT), which translates CT into synthetic MR before localisation. On an external HCC cohort, synthetic transfer outperformed other tested methods and CT-only baselines. Our framework enables cross-modality tumour localisation without paired MR-CT training images.