Reinforcement learning affects performance of chest x ray report checks
Unmentioned Checklist Findings Change How Reinforcement Learning Appears to Improve Chest Radiograph Report Checking
Computer Vision and Pattern RecognitionComputation and LanguageMachine Learning
Summary
Checking medical reports automatically can miss certain findings if those findings aren't mentioned. The authors trained a computer model using reinforcement learning to fill out a checklist of possible problems from chest x-rays, then checked report sentences against this list. They found that how unmentioned findings are treated during training changes the model's apparent accuracy, and different checking methods vary greatly in accepting statements about unmentioned findings. This work shows that evaluation details can influence how much reinforcement learning seems to help in verifying radiology reports.
What this means in practice
- •For hospital data teams: Improve automatic detection of unmentioned findings in chest x-ray reports to reduce false alarms and better evaluate AI checkers.
- •For radiology software developers: Design radiology report checking systems that carefully handle unmentioned findings to avoid misleading performance gains from reinforcement learning.
Authors
Ali Vosoughi, Akhil Kasturi, Chenliang Xu, Axel Wismueller
Abstract
Automated checks of radiology reports may rely on AI-generated checklists that leave findings unmentioned. We used reinforcement learning to train a vision-language model to fill in a 12-finding checklist from a chest radiograph without seeing the sentence under test; a separate checking model judged the sentence from the checklist. On held-out patients, a rule-based check and an independent medical checker, neither used in training, measured discrimination gains (Youden index) of 12.6% and 11.8%; only the rule-based check met the prespecified false-alarm criterion. Switching to the training format, which fixes finding order and enters unmentioned findings as absent, raised the training checker's measured gain and lowered the independent checker's, a prespecified comparison that yielded 6.2% (95% interval 2.0% to 10.5%) and, post hoc on held-out patients, 7.7%. Across 8 checking models, acceptance of a label-consistent negative statement about an unmentioned finding ranged from 1.0% to 97.0%. Labels were report-derived, not radiologist-adjudicated.