Multimodal models struggle to predict evolving mental states in social interactions
Can You Predict the Next Mind? Toward Multi-Turn Multi-Party Mental World Modeling
Human-Computer Interaction
Summary
Computers have gotten good at understanding and interacting with the physical world, but predicting what people think and feel over time during conversations remains difficult. The authors created a new test and a large dataset to see how well AI can guess someone’s future thoughts and actions in multi-person interactions. They found current AI systems often fail at this task because they don’t model how mental states change over time. This research helps identify where AI needs improvement to better understand and adapt in social situations.
What this means in practice
- •For chatbot developers: Build conversational agents that anticipate user emotions and intentions over multiple messages to improve responsiveness.
- •For social robotics teams: Develop robots that predict human partners’ mental states over extended interactions for more natural social behaviors.
Authors
Jiahao Huang, Zheng Lian
Abstract
While multimodal foundation models have achieved remarkable progress in physical world modeling, generation, and embodied reasoning, their capacity for dynamic mental world modeling remains underexplored. Existing benchmarks predominantly focus on static mental state perception or short-horizon decision-making, overlooking the dynamic mental trajectories inherent in multi-party, multi-turn interactions. To bridge this gap, we introduce M$^3$WME-Bench, a benchmark designed to evaluate mental world modeling conditioned on the evolving actions of interactants. Specifically, models are tasked with forecasting a target subject's future mental states and behaviors over extended social interactions. In addition to the benchmark suite, we construct M$^3$WME-23K, a dedicated training dataset, and establish a competitive baseline named MindNext-7B. Extensive evaluations across four representative paradigms (general MLLMs, emotion-specialized MLLMs, agentic frameworks, and World Foundation Models) reveal that current systems consistently struggle to anticipate forward-looking mental states. Mechanistic analyses further reveal that existing architectures fundamentally lack forward mental state transition dynamics. Our findings highlight a pivotal challenge in advancing agents from physical to mental world modeling, charting new directions for socially adaptive AI.