Clinical AI models reveal bigger hidden subgroup biases in patient care
Detection and Validation of Hidden Subgroup Disparities in Clinical AI
Machine Learning
Summary
AI tools used in healthcare can sometimes work better for some groups of patients than others, which is a problem for fairness and safety. The authors showed that traditional ways to find these differences don’t capture all the important groups, especially when patient details are missing. They used a new method that finds hidden patient groups based on the AI’s own data and found these groups often had bigger drops in accuracy. They also showed these hidden groups could be explained using known patient or imaging info and used to reduce bias. This approach can help hospitals better check and improve their AI tools.
What this means in practice
- •For hospital data teams: Detect hidden patient groups with different AI accuracy to improve clinical model safety and fairness.
- •For medical imaging engineers: Use learned subgroup annotations from chest X-rays to identify and mitigate bias in AI diagnostic models.
Tested on one dataset.
Authors
Alceu Bissoto, Trung-Dung Hoang, Tim Flühmann, Nina Weng, Susu Sun, Christian F. Baumgartner, Lisa M. Koch
Abstract
Machine learning (ML) models used in clinical workflows often perform systematically differently across patient groups, raising concerns about equity and patient safety. Traditional subgroup analyses can expose such disparities only when rich metadata (e.g., patient demographics) is available and captures the sources of performance variation, conditions rarely met in practice. Algorithmic subgroup discovery methods offer a solution by identifying coherent groups from learned feature representations, but because true underlying stratifications are unknown, these methods are hard to evaluate. As a result, subgroup discovery has not been systematically applied or assessed for performance monitoring. Our main objective is to evaluate algorithmic subgroup discovery for performance monitoring, assessing the discovered subgroups' performance gaps and semantic coherence. Investigating cardiomegaly classification in a dataset of 223,228 chest X-rays, we showed that the discovered subgroups exhibited larger performance gaps (accuracies of 0.61-0.90) than traditional metadata-based analysis (0.80-0.94) and were frequently interpretable using patient, acquisition, and model metadata. As a proof of concept, we also showed that annotations from learned subgroups can be used to mitigate the biases they expose. We conclude that subgroup discovery is a practical and effective tool to enhance the performance validation and monitoring of ML systems in healthcare.