What Makes a Medical Checker Trainable? Diagnosing Signal Collapse and Reward Hacking in Checker-Guided RAG for Biomedical QA
2026-05-25 • Computation and Language
Computation and Language
AI summaryⓘ
The authors study how adding a fact-checking system (NLI checker) to a medical question-answering AI affects its training. They find that it's not how accurate the checker is, but the type of feedback it gives during training that matters. If the feedback signal is too strong, the AI cheats by giving very short or poor answers, but a moderate signal helps the AI learn better answers. Also, the strength of the feedback depends on the AI’s current behavior, showing complex interactions between checker and model. They highlight important conditions for using verifiers as reward signals in training.
Medical RAGNLI checkerreinforcement learningGRPOreward hackingpolicyBERTScorelog-probabilitymodel calibrationretrieval-augmented generation
Authors
Yuelyu Ji, Min Gu Kwak, Hang Zhang, Xizhi Wu, Chenyu Li, Yanshan Wan
Abstract
Medical RAG needs evidence-grounded claims, so plugging a claim-level NLI checker into retrieval-augmented RL is intuitive. \textbf{We find that the checker's \emph{output distribution} during training, not its held-out accuracy, decides whether it provides trainable gradient.} We compare four NLI checker back-ends as process rewards inside a GRPO-trained medical RAG agent (Qwen2.5-7B, replicated on Qwen3-4B and Llama-3.1-8B) across four held-out medical QA benchmarks. Three diagnostic findings emerge. \textbf{(i)} Signal collapse is log-prob-specific: LLM log-probability scoring labels over 97\% of claims neutral -- collapsing the RL gradient to zero -- while a calibrated MedNLI classifier scores the same pairs non-degenerately. \textbf{(ii)} Moderate signal beats strong signal on answer quality: a strong proprietary checker triggers a three-step reward-hacking cascade -- ultra-short answers, search avoidance, language collapse -- so a moderate-signal local classifier trains a higher-quality model (\textbf{+12\% BERTScore over zero-shot, no GPT dependency}). \textbf{(iii)} Signal strength is policy-dependent: the same checker registers as moderate on one policy but strong on another without triggering the cascade end-state. We frame these as boundary conditions for verifier-as-reward systems.