Radiology report style affects AI evaluation results in chest X-rays
Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation
Computation and LanguageArtificial Intelligence
Summary
Radiologists write reports on chest X-rays in many different styles, even when they see the same problems. This makes it hard to judge how well AI tools generate such reports because the AI is compared against human reports that might use different words or formats. The authors show that small changes in the style of the reference reports can change which AI model looks best. They created a method to rewrite reports in different styles without changing the medical meaning and released a dataset to help future work on this.
What this means in practice
- •For radiology ai developers: Choose or create reference reports that match desired reporting styles to fairly evaluate and compare AI radiology report generators.
- •For medical data annotation teams: Use the ReRef method to produce multiple stylistic variants of radiology reports for training and testing AI systems under different reporting conventions.
Tested on one dataset.
Authors
Daniel P. Jeong, Charles Q. Li, Hossein Hosseiny, Nitya M. Bhalla, Fatma Uyar Morency, Pradeep Ravikumar, Zachary C. Lipton, Michael Oberst
Abstract
Radiologists follow heterogeneous reporting practices. Two radiologists examining the same image and identifying the same clinical findings might nevertheless compose superficially distinct reports, varying in terminology, shorthand, formatting, and level of detail. These variations in reporting norms represent an under-appreciated obstacle in efforts to evaluate AI-based radiology report generation (RRG) models, where machine-generated reports are typically assessed based on their concordance with human-generated references. In this paper, we quantify the sensitivity of established evaluation metrics to variations in reporting practices, revealing impacts large enough to alter the rankings of models. We introduce a radiologist-informed taxonomy of variations in radiology reporting practice and a method (ReRef) that rewrites reference reports along the axes of our taxonomy while preserving clinical interpretation. For instance, when comparing the performance of nine RRG models on MIMIC-CXR using RadCliQ-v1, condensing the discussion of normal findings in the reference reports causes Libra to drop from first to second place while CheXOne rises from third to first. Our results suggest that many current metrics fail to decouple clinical interpretation from conformity to reporting practices and that choosing the ``right'' references that accurately reflect the desired reporting practices can be important in practice. To support future research, we release MIMIC-CXR-Ext-ReRef, a radiologist-validated dataset of 120 (original, alternative) reference report pairs derived from MIMIC-CXR.