Language model agents read minds poorly despite fluent talks
Mind or Message? Auditing Theory of Mind in Multi-Agent Social Simulation
Machine Learning
Summary
The paper studies whether AI agents truly understand their conversation partners or just mimic their words. The authors built a negotiation simulation where they know exactly what each side wants and measured how well agents predict each other's preferences. They find agents are socially smooth but often fail to reach the best possible deals and misunderstand true partner goals. Agents rely mostly on their own views rather than accurately modeling the other’s mind.
What this means in practice
- •For chatbot developers: Improve understanding of current limitations in agents’ reasoning about user goals during multi-turn conversations.
- •For game ai designers: Design agents that negotiate and interact more authentically by recognizing agents’ tendency to misunderstand partners’ preferences.
Authors
Cong Li, Cheng Chen, Thomas Fung, Alex Rossi, Yi Li
Abstract
Language model agents are increasingly used to simulate social interaction, and the resulting transcripts read as though the agents understand one another. We ask whether that appearance rests on a model of the partner's mind or on the surface record of what the partner said. We build a social simulation in which both questions have exact answers: 40 multi-issue negotiations whose hidden preference weights and whose full Pareto frontier are known by construction. Two model families negotiate across 160 dyads, every transcript is frozen before any measurement, and 2880 counterfactual probes then hold the evidence byte identical while moving one factor at a time: the reader's own stake, the partner's tone, an identity label, and the order of recursion. The agents are socially fluent and economically poor. They reach agreement in 96.2% of dyads with 0 protocol failures, yet only 0.7% of deals land on the Pareto frontier, they leave 20.5% of the available joint value unclaimed, and they miss the one issue on which their interests are perfectly aligned in 76.6% of deals; on the frontier and on that aligned issue, a package drawn at random from the set both sides would accept does as well. The probes locate the failure. Swapping only the reader's own payoff sheet, while the partner's words and offers stay identical, moves the inferred top priority by 15.0 percentage points, which is egocentric projection rather than inference, while a tone rewrite moves it by 5.3 percentage points and an identity label by 0.0. Most tellingly, an agent predicts what its partner believes about it 72.5% of the time while that partner's belief is itself correct only 51.2% of the time: the agents track the conversation far better than they track the mind behind it.