Rag models often cite wrong sources despite training for accuracy
Attributable Post-Rationalization in RAG Citations: A Controlled Reproduction and an RLVR Comparison
Computation and LanguageInformation Retrieval
Summary
Some AI systems answer questions and include sources that they didn’t actually use, a behavior called post-rationalization. The researchers tested whether training these systems to be accurate also makes their citations honest. They found that even with special training focused on correctness, the systems still often cite wrong or unrelated sources. This means that simply rewarding correct answers isn’t enough to ensure truthful citations, which must be addressed separately.
What this means in practice
- •For ai development teams: Improve accuracy metrics by separately training models to produce truthful citations beyond just correct answers.
- •For content moderation teams: Detect and flag AI-generated citations that may not be reliable to reduce misinformation in automated content.
Authors
Mehedi Khan, Md. Shariful Islam Bhuyan
Abstract
A RAG system can hand you the right answer and cite a source it did not actually use. Models output these unfaithful citations via post-rationalization: they write the answer first and then attach a citation to whatever passage looks close enough. Search agents are now trained with reinforcement learning from verifiable rewards (RLVR), which pays them for getting the answer right. We asked whether that training also teaches them to cite honestly. Improving an existing methodology with a required control, we compared an instruction-tuned model against three RLVR agents trained from it, on four question-answering datasets, using only free-tier Kaggle GPUs. Post-rationalization is everywhere: on Wikipedia-based questions roughly one citation in seven is unfaithful. RLVR does not fix it. The agents post-rationalize at their base model's rate, and one lands slightly worse. Rewarding correct answers buys nothing in citation faithfulness, so faithfulness has to be trained and measured on its own terms.