Specialized foundation models generate better chest x ray counterfactual images
Generating Chest X-Ray Counterfactuals by Specialising Foundation Image Models
Computer Vision and Pattern Recognition
Summary
Chest X-rays are important for diagnosing diseases, but it's hard to imagine how an X-ray might look under different hypothetical conditions. The authors developed a method to adapt existing general image models to produce these 'what if' images for chest X-rays, even when data is limited or different from training examples. Their system, RadCF, creates realistic new X-ray images that help understand changes and improve medical classifiers. This approach could help doctors and AI systems think about alternative scenarios in medical imaging.
What this means in practice
- •For radiology ai developers: Improve robustness of AI diagnostic models by using generated counterfactual chest X-rays to detect and reduce shortcut learning.
- •For medical image software engineers: Create tools that can simulate alternative disease scenarios in chest X-rays to aid training and validation of medical imaging systems.
- •For healthcare software product teams: Develop enhanced diagnostic imaging products that generate hypothetical patient scans for decision support and clinical explanation purposes.$Commercial implications: This paper enables practical counterfactual chest X-ray generation, which could be incorporated into diagnostic software sold to hospitals and clinics.
Authors
Xiaodan Xing, Rajat R. Rasal, Julia A. Meister, Sara Ghorayeb, Galvin Khara, Jessica Schrouff
Abstract
Counterfactual image generation answers questions about how a subject would have looked under retrospective, hypothetical scenarios. Recent methods have improved perceptual quality, identity preservation and faithfulness to an underlying causal model, but their adoption in healthcare is limited by scarce annotated data, distribution shift between datasets, and mismatches between pretrained generative models and those required for counterfactual inference. We propose specialisation, a data and parameter-efficient framework for adapting pretrained, non-causal generative models into causal mechanisms under distribution shift. Based on this framework, we train a radiology counterfactual image generation model, called RadCF, using latent flow matching. We validate our approach on three chest X-ray datasets spanning different dataset shifts, data volumes, and counterfactual questions, associated with challenging, highly-localised interventions. Our results show that RadCF and specialisation improve counterfactual soundness over existing methods while being data and parameter efficient, and that the resulting counterfactuals can detect and mitigate shortcut learning in a downstream medical classifier. Code is available at https://github.com/GSK-AI/RadCF/.