Language models rarely know what they themselves would do
Strangers to Themselves: What Language Models Say About Themselves Is Generic
Machine LearningArtificial IntelligenceComputation and LanguageComputer Vision and Pattern RecognitionComputers and Society
Summary
Language models can describe how they might behave in different situations, but this research shows that their self-descriptions are not very accurate. The authors found that asking a model about itself is no better at predicting its behavior than asking about a generic AI or even other models. Models also tend to paint themselves in a more positive light. This means that language models' self-reports mostly reflect general ideas about AI assistants, not true self-knowledge.
What this means in practice
- •For ai system developers: Avoid relying on language models’ self-reports to predict their behavior or risks when designing AI assistants.
- •For chatbot trainers: Use behavioral records rather than self-descriptions to improve chatbot predictions and fine-tune responses more reliably.
Authors
Phil Blandfort, Urja Pawar
Abstract
Language models can fluently describe how they would behave: whether they would cave to pushback, misuse a tool, or lie under pressure. Is that description actually about the model speaking? We turn self-knowledge into a prediction test. Across nine behavioral evaluations, we measure how a model behaves under different conditions, ask it to predict those rates, and compare its predictions with controls that remove the self from the question. We find that: (i) Direct self-report is weak (r = +0.04), and even showing the model the exact items only raises prediction to +0.24. Crucially, the same item-informed question about "capable AI agents in general" does just as well (+0.28), while other models' answers about themselves predict the target model at least as well as its own. (ii) Frontier scale does not detectably change this pattern: any gains in prediction are not self-specific, and are consistent with a better theory of how AI assistants behave rather than better self-knowledge. (iii) First-person framing does have one robust effect: it shifts reports in the flattering direction, understating harmful behavior relative to the same question about a generic agent. (iv) Finetuning on a model's own behavioral record can teach narrow self-predictions, but it also changes the behavior being predicted and the gains do not transfer broadly. The practical implication is simple: asking a model what it would do mostly reveals a theory of AI assistants in general, plus a favorable bias, rather than privileged knowledge of that model.