Counterfactual training helps language models admit when info shapes answers
Did the Model Use That? Counterfactual Training Boosts Self-Attribution in Language Models
Computation and LanguageArtificial Intelligence
Summary
It can be hard to tell if a language model actually used extra information given with a question when it answers. This paper looks at whether models admit when their answers depend on that extra information and find most models don’t do this well. The authors introduce a new training method called CREDIT that teaches models to say clearly when their answers rely on the provided info. CREDIT improves models’ honesty about where their answers come from without needing humans to label data.
What this means in practice
- •For ai developers: Train language models to clearly indicate when answers are influenced by provided information, improving trust and accountability in AI responses.
- •For content moderation teams: Use models with improved self-attribution to better verify sources influencing generated content and detect misleading AI outputs.
Authors
Zikai Zhou, Jiacheng Sang, Yilin Xu, Yufei Jin, Monica S. Lam
Abstract
A fundamental challenge for explainable and accountable language models is revealing how external information influences their answers. This paper examines whether models are self-attributive: that is, whether they indicate when -- and only when -- their responses to questions are influenced by information provided alongside the question. Furthermore, we explore how self-attribution can be improved through counterfactual training, which explicitly teaches models to distinguish between answers that do and do not depend on supplied information. We found that most of the eleven frontier models rarely attribute influential information in their answers, and the rest attribute non-influential information. Training acknowledgment into reasoning does not fix this: the acknowledgment does not reach the final answer. We introduce CREDIT, which directly supervises final-answer self-attribution with counterfactual labels, without the need for human annotation. CREDIT substantially improves attribution in four open models, including two distilled reasoning models, and outperforms reasoning-level fine-tuning and prompting baselines. Among answers that agree with a user's suggestion, CREDIT distinguishes those changed by the suggestion from those the model would have given anyway. Gains transfer partly to unseen information types and prompts containing sources that support conflicting answers.