Image to video models struggle matching actions to mental states
MindWorldBench: Evaluating Mental-State-to-Behavior Reasoning in Image-to-Video Generation
Computer Vision and Pattern Recognition
Summary
Current models that turn images into videos can make realistic and physically plausible scenes, but they cannot understand or show what people are thinking or feeling. The authors created MindWorldBench, a way to test if models can guess actions based on invisible mental states like beliefs or desires without being told the actions directly. Their tests showed that while videos look good, models usually ignore what a person thinks or believes and instead act like they know everything perfectly. This points to a need for better ways to teach machines about people's mental states to improve video generation.
What this means in practice
- •For video game developers: Improve non-player character behavior in games by integrating mental-state conditioned actions for more believable interactions that account for subjective beliefs.$Commercial implications: Enables creation of video game characters with actions influenced by mental states, enhancing player immersion and engagement.
- •For ai video tool makers: Create video generation tools that better reflect human mental states when generating behavior sequences from images without explicit action input.
Authors
Ruiqi Li, Xuanyi Liu, Sijia Li, Haofeng Wang, Yuxin Liu, Feng Xie, Songchao Tan, Shiqi Wang, Hanwei Zhu, Yizong Wang, Chuanmin Jia, Siwei Ma
Abstract
Current image-to-video models achieve visual realism and physical plausibility, but reasoning about mental states remains unexplored. Actions are driven by belief, desire, and perception, requiring inference beyond explicit instructions. We introduce MindWorldBench to evaluate mental-state-conditioned video generation. We formalize this as mental-state-to-behavior reasoning, where models generate actions from a world state and latent variables without explicit action prompts. MindWorldBench utilizes Zero-Action Prompting and a counterfactual design with 744 prompts to isolate the causal effects of mental states. An automated pipeline evaluates video quality, commonsense plausibility, and mental-state consistency. Evaluations of 11 models show that despite visual fidelity and physical reasoning, models fail to align behaviors with latent mental states. We identify a failure mode, termed Omniscient Bias, where models default to the objective world state rather than human's subjective belief. These results demonstrate a disconnect between visual generation and cognitive reasoning, suggesting a need for explicit mental-state modeling in video generation systems. Project website: https://richard2049-lee.github.io/MindWorldBench/