Language model self-reports do not prove the presence of self awareness
Self-Reports Do Not Identify Self-Models: An Identifiability Test for Counterfactual Reports
Computation and Language
Summary
This paper shows that when language models say what they would do in different situations, it doesn’t necessarily mean they have a real understanding or self-awareness. The researchers tested if the models’ statements about feelings would stick to the same cause when the setting changed, and found they often don’t. This means using these self-reports as proof that language models have a sense of self is unreliable.
What this means in practice
- •For ai system developers: Design evaluation benchmarks for language models that test stability of self-reports under changing contexts to better assess internal mechanisms.
- •For machine learning evaluators: Use environment-shift invariance tests before accepting language model self-report accuracy as evidence of autonomous reasoning.
Authors
Phongsakon Mark Konrad, Toygar Tanyel, Serkan Ayvaz
Abstract
Language-model self-reports are evidence about behavior in a prompt environment, not by themselves evidence of a self-model. We investigate counterfactual reports about affect-like states under activation interventions and ask whether the report remains bound to the named intervention when the demonstration environment changes. Across three open instruction models, wrong-source demonstrations move reports toward the source answer family, while explicit mechanism binding reduces this pull. Self-report benchmarks should include environment-shift invariance tests under fixed intervention before treating accuracy as evidence for an autonomous report mechanism.