Test-time augmentation improves calibration of black-box radiology ai
Improving Calibration of Black-Box Radiology AI Using Test-Time Augmentation
Machine LearningHuman-Computer Interaction
Summary
Radiology AI systems help doctors make important decisions but need to be accurate about how certain they are. The authors found a way to make these AI systems better calibrated without changing the AI itself, by applying smart changes to the images they analyze and combining the results. This method, called DualTTA, works even when the AI is a "black box" and improves the AI's confidence accuracy in detecting health issues like blood clots and brain bleeding. It also performs better than some methods that require access to the AI’s internal settings.
What this means in practice
- •For hospital ai deployment teams: Enhance confidence reliability of proprietary radiology AI systems without access to model internals by applying test-time augmentation during inference.
- •For medical device integrators: Improve calibration of embedded black-box clinical AI products using learned aggregation of augmented input images at inference.$Commercial implications: Enables adding calibration improvement features to commercial AI-powered imaging devices without retraining or model access.
Authors
Nathan Le, Magdalini Paschali, Arogya Koirala, Andrew Johnston, Zhongnan Fang, David B. Larson, Akshay S. Chaudhari, Camila Gonzalez
Abstract
Radiology AI systems increasingly inform clinical decisions such as triage, follow-up imaging, and treatment planning. For these decisions to be made safely, model outputs must be well calibrated, meaning predicted probabilities accurately reflect true risk. Many standard techniques for improving calibration, such as MC Dropout and Deep Ensembles, require access to model parameters or retraining. However, proprietary clinical AI systems operate as black boxes, preventing access to the model's internals. To that end, we propose a model-agnostic framework for improving calibration of black-box models using clinically grounded test-time augmentation (TTA). Our framework applies geometric and physics-inspired 3D CT perturbations and learns probability-level aggregation strategies without access to model internals or the original training data. Across pulmonary embolism and intracranial hemorrhage detection tasks, DualTTA achieved the strongest overall calibration among TTA methods, reducing the Expected Calibration Error by 54% (0.239 -> 0.109) and 43% (0.051 -> 0.029), respectively, while requiring only input-output access. Additionally, DualTTA outperformed uncertainty estimation techniques that require access to model internals, such as Temperature Scaling, MC Dropout, and Deep Ensembles, in most calibration metrics. These results demonstrate that learned TTA aggregation can improve the calibration of clinical AI systems, providing a practical approach for improving the reliability of black-box medical AI.