REVA-PO: Stabilizing Reinforcement Learning for Chest X-ray Report Generation
2026-07-11 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors developed a new method called REVA-PO to improve how computers write chest X-ray reports by learning over time. Their approach adjusts how much the computer sticks to previous good behaviors based on how well it's doing, and it also resets its learning periodically to avoid getting stuck. They tested this on two big chest X-ray datasets and found their method made reports that were more accurate and better written compared to past approaches.
reinforcement learningKullback-Leibler regularizationpolicy optimizationchest X-ray report generationlarge language modelsBLEU scoreCheXpertRadGraphsupervised fine-tuningMIMIC-CXR dataset
Authors
Li Guo, Anas M. Tahir, Z. Jane Wang
Abstract
Automated chest X-ray report generation has recently benefited from reinforcement learning (RL) and large language models. However, RL training often suffers from instability or limited exploration due to fixed Kullback-Leibler (KL) regularization and a static reference policy that accumulates KL pressure over time. We propose Response-Weighted and Validation-Anchored Policy Optimization (REVA-PO), a RL framework that stabilizes long-term training via Response-Weighted Regularization (RER) and Validation-Anchored Policy Reset (VAPR). RER dynamically adjusts per-response KL weights based on advantage and reference-policy entropy, relaxing constraints for high-quality responses while tightening them for low-quality ones. Complementarily, VAPR periodically synchronizes the reference and current policies to the best validation checkpoint, resetting accumulated regularization pressure to expand the viable exploration space. To ensure a robust starting point, we employ a three-stage pipeline consisting of warm-up training, classifier-guided supervised fine-tuning, and RL. Extensive evaluations on MIMIC-CXR and IU-Xray demonstrate that REVA-PO sets new state-of-the-art benchmarks in both linguistic quality and clinical accuracy. Notably, BLEU-4 improves by 5.1% on MIMIC-CXR and 3.6% on IU-Xray, while CheXpert F1 and RadGraph F1 scores increase by 4.5% and 12.8%, respectively, over prior leading methods. The code is publicly available at https://github.com/LiGuo12/REVA_PO/.