Better AI understanding of others by tracking what they know
Who Knows What - and What Does It Support? Perspective-Bounded Evidence Calibration for Multimodal Theory of Mind
Artificial Intelligence
Summary
Understanding what another person knows or believes is very important for social interactions. Current AI models often confuse what an agent has actually seen or heard with information only the AI has, leading to mistakes in guessing that agent's thoughts. The authors introduced a method called PIP-ToM that carefully tracks what evidence is available to each agent and uses that information to improve belief and intention prediction. Their method improves the accuracy of predicting others' mental states in different tests compared to previous approaches.
What this means in practice
- •For conversational ai developers: Improve chatbots’ ability to understand and predict user beliefs and intentions by limiting inference to user-accessible evidence.
- •For interactive game designers: Create more realistic non-player characters that infer others’ beliefs and goals based only on their in-game observations.
Authors
Ziqi Liao, Jiaqi Shen, Yiming Na, Kedi Lyu, Shao-Yuan Lo, Hongxia Xie
Abstract
Theory of Mind (ToM), the ability to infer others' beliefs, goals, and intentions, is a cornerstone of social intelligence. Multimodal ToM requires inferring an agent's mental state from evidence that is available to that agent, rather than from the complete episode observed by the solver. Existing LLM-based and Bayesian model-based approaches reconstruct global event traces, yet they do not explicitly distinguish objective events from agent-admissible evidence during hypothesis evaluation. Consequently, agent-inaccessible events or semantically misaligned actions and utterances may influence the inferred belief, goal, or social intention, leading to perspective leakage and source or route confusion. We formulate this limitation as an evidence-admissibility problem: evidence should contribute to a mental-state hypothesis only when it is accessible to the relevant agent, preserves its information source, and is diagnostic of the queried mental-state variable. To operationalize this principle, we propose PIP-ToM, an evidence-admissibility interface that constructs agent-specific evidence stores, aligns them with structured mental-state claims, and calibrates model-based hypothesis scores while preserving the underlying Bayesian inference process. PIP-ToM with GPT-4o outperforms existing ToM methods across MuMA-ToM, MMToM-QA, and Hi-ToM, achieving average accuracies of 92.56%, 95.00%, and 95.00%, respectively.