Summary
Medical AI models used to detect tuberculosis from chest X-rays do not always perform the same across different patient groups and testing conditions. The researchers tested three specialized models and one general model on over 12,000 X-rays from several hospitals. They found that the reliability and accuracy of these models depend heavily on factors like the types of controls used, the prompts given to the models, and the choice of scoring thresholds. This means that a model’s reported performance in one setting may not hold true in another, so it’s important to carefully check how these models are evaluated before trusting their predictions in clinics.
What this means in practice
- •For hospital data teams: Ensure reliable tuberculosis screening by verifying model performance varies with patient groups and test conditions before deployment.
- •For medical ai developers: Test vision-language models on diverse cohorts and prompts to confirm stable tuberculosis detection before clinical use.
Abstract
A medical model's benchmark score does not establish that the same conclusion holds under a different evaluation. This study tests whether claims about model ranking, score reliability and screening performance survive changes in cohort, prompt, negative spectrum, specified prevalence and operating threshold. We audit three medical vision-language models (BioMedCLIP, CheXficient, and MedSigLIP) and a general-domain OpenCLIP comparator on 12,200 chest radiograph records from four datasets (Montgomery, Shenzhen, TBX11K, and VinDr-CXR). Five fixed prompt families yield 244,000 model--image--prompt scores. No model leads every cohort and reliability criterion. Prompt-family changes alter AUROC in 21 of 48 multiplicity-controlled comparisons. Replacing healthy controls with sick non-tuberculosis controls reduces AUROC by 0.075--0.306 across all four models. On VinDr-CXR, the three medical models distinguish tuberculosis from no-finding controls substantially better than from pneumonia or lung tumor; their AUROC point estimates for both named diseases fall below 0.5. CheXficient has documented VinDr-CXR pretraining exposure, which limits the interpretation of its results. Thresholds chosen for 95\% sensitivity on TBX11K training retain that constraint by point estimate in only four of sixteen target evaluations. A five-seed supervised source model reaches 0.999 AUROC on TBX11K validation but 0.629 on each of two external cohorts. Conservative exclusion of perceptual-overlap candidates narrows this gap without closing it. These retrospective, single-task results show that discrimination, score reliability and threshold retention support different portability claims. Evidence for chest X-ray tuberculosis screening should identify the complete evaluation specification rather than attribute clinical portability to a checkpoint alone.