Learning to defer improves ai and human teamwork on chest x-rays
Learning to Defer with Guidance on Real World Medical Data
Machine Learning
Summary
Reading and interpreting medical images takes a lot of time and effort. AI can help but sometimes it makes mistakes or misses details, which can cause extra work for doctors. The authors studied a method called Learning to Defer (L2D), which lets AI decide when to make its own diagnosis and when to ask a human expert for help. They tested this method on real chest X-ray data with multiple doctors’ opinions and introduced an improved approach called L2D with Guidance, where AI also offers advice when deferring. Their results show this new method works better than AI alone, human alone, or previous deferring methods.
What this means in practice
- •For radiology teams: Reduce radiologist workload by routing chest X-ray cases adaptively between AI-alone analysis, human review, or AI-guided human interpretation based on case difficulty.
- •For medical imaging software developers: Incorporate a three-way deferral model that enables AI to either predict, refer, or assist experts in real-world diagnostic tools for chest radiographs.$Commercial implications: Enables development of advanced diagnostic software combining AI predictions with expert input and AI guidance for hospitals.
Authors
Emma Sun, Joshua Strong, Alison Noble
Abstract
Medical image interpretation is high-volume and time-consuming, and while AI interpretation can reduce workload, fully autonomous deployment carries potential safety concerns and low specificity may in practice lead to increased clinician workload. Learning to Defer (L2D) addresses this by selectively routing cases between autonomous prediction and human experts by learning from input features and AI model and human performance. While theoretical guarantees have been proven for L2D, its performance has not been validated on real-world medical datasets with human reader annotations. We evaluate the predictor-rejector formulation of two-stage L2D, where the AI predictor model is fixed and separate from the trainable routing or rejector model, on Collab-CXR, a multilabel chest X-ray dataset with multiple human annotations per case. This is the first work to look at L2D in the context of real-world medical imaging data with human annotations. We further introduce a new setup, L2D with Guidance, where the decision space is extended to three choices: predict autonomously, defer to a human expert, or defer to a human expert and provide AI guidance. We compare multiple rejector architectures and loss functions, and different input feature availabilities. This is reproduced on two larger datasets, VinDr-CXR and CheXpert. Our results show that two-stage L2D with Guidance outperforms classic two-stage learning to defer, as well as human-alone, AI-alone and AI-guided human baselines. Notably, this performance is achieved with simpler loss functions compared to formally defined L2D surrogate loss functions in current literature.