Evaluating and Improving LLM Self-Modeling
2026-08-31 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors investigate how well large language models (LLMs) can understand and explain their own behavior, especially when asked if changing a prompt would alter their answers. They created a benchmark to test these self-understanding skills and found that current models have limited ability and often make mistakes on simple 'what if' questions. To improve this, the authors developed a method to generate training data for self-modeling and used reinforcement learning to boost performance across several open-source models. However, these improvements do not necessarily mean the models gain true introspection or real insight into their internal decision-making.
Large Language ModelsSelf-modelingReinforcement LearningBenchmarkingPrompt EditingCounterfactual QuestionsSynthetic DataIntrospectionOpen-source Models
Authors
Siqi Zeng, Andre N. Assis, Rowan Wang
Abstract
We study self-modeling: an LLM's ability to answer questions about its own behavior. We focus on verifiable behavioral questions, such as whether a prompt edit would change the model's final answer. To measure this capability, we introduce a benchmark that tests diverse types of self-modeling questions. Current models show non-trivial but limited self-modeling skill, and make systematic mistakes on simple counterfactual questions about their own behavior. To improve self-modeling skill, we develop a scalable synthetic-data pipeline that produces self-modeling training data, and show that reinforcement-learning can improve aggregate self-modeling skill across three open-source model families with some transfer to held-out tasks. These gains, however, do not seem to constitute introspection consistently: improved self-modeling may not arise from privileged access to the model's internal decision process.