Jev system judges factual accuracy in AI radiology reports
Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factuality
Computation and LanguageArtificial Intelligence
Summary
AI can write radiology reports that look real but sometimes leave out important problems or add mistakes. The authors studied a tool called Jev that quickly checks if one report agrees with another to spot these errors. Jev compares statements from both reports to find missing or unsupported findings and works well compared to other methods. It can do this affordably and with fewer questions, helping improve how AI-generated reports are evaluated. The authors also tested Jev's ability to spot a specific error type, showing it can reliably detect certain mistakes.
What this means in practice
- •For medical ai developers: Integrate Jev to evaluate AI-generated radiology reports efficiently by checking factual consistency against expert reports.
- •For hospital data teams: Use Jev to monitor and ensure the factual reliability of automatically generated clinical radiology reports during deployment.
Authors
Jiaju Huang, Hao Yang, Xinyu Ma, Xinglong Liang, Kunyan Cai, Junqiang Ma, Shaobin Chen, Yue Sun, Tao Tan
Abstract
An AI-generated radiology report can resemble a physician's report while omitting an abnormality, adding an unsupported finding, or reversing its presence. Measuring these factual differences is essential for evaluating report generators. We study Jev, a System One decision model, as a simple, low-cost judge of agreement with physician-written reference reports. Our evaluator checks whether each statement is supported by the other report and combines these judgments in both directions to capture unsupported claims and omissions. A single-question configuration reaches Kendall correlations of 0.573 on RadEvalX and 0.398 on RadEvalExpert with expert error counts, outperforming an open natural language inference judge under matched decomposition and aggregation. One support question per statement retains similar expert agreement to seven while using 43-45% fewer judgment input tokens. At the documented API price, judgments cost under three cents per hundred report pairs, excluding local decomposition. In a separate controlled-error test, Jev detects false negation with an AUROC of 0.977. Local RadMatch achieves stronger agreement on clinically significant errors in both expert datasets and on total errors in the shared RadEvalExpert subset. Finding-count and error-scope analyses show that benchmark agreement reflects report size and error definitions as well as medical error detection. These results support Jev as a practical judgment component for measuring factual differences in generated radiology reports and identify where more elaborate evaluation remains valuable.