Psychology informed benchmark improves video social understanding tests
PsycheMME: A Psychology-Informed Benchmark for Social Understanding and Interaction in Videos
Computer Vision and Pattern Recognition
Summary
Understanding social situations in videos involves figuring out what people want and how they should respond, which are different skills. The authors created PsycheMME, a dataset and test that focuses on both understanding and decision-making, using ideas from psychology. It includes questions about human states, needs, memory, and interactions, and checks answers by how well their explanations match evidence. Testing 17 AI models showed where they succeed or fail in grasping social cues and making good choices.
What this means in practice
- •For ai system developers: Evaluate AI models’ understanding and response selection in social video scenarios using psychology-informed benchmarks.
- •For interactive media creators: Improve social interaction realism in video content by testing AI-driven characters with nuanced psychology-based evaluation questions.
Authors
Heran Hu, Yuhang Dong, Zhucun Xue, Kaiwei Chen, Yong Liu, Jiangning Zhang
Abstract
Understanding human needs and selecting responses require distinct judgments that aggregate accuracy can obscure. We introduce PsycheMME, a video benchmark informed by psychology, with eight capabilities spanning grounding, states, needs, interaction decisions, memory, and risk. Drawing on Social Information Processing and the Perception-Action Cycle, this organization separates human understanding from response selection. We construct 1,000 questions from 630 episodes through independent answering with proposed references withheld and expert review. Our Reasoning-Verified protocol weights answer credit by the support and logical consistency of explicit rationales against annotated cues. Across 17 models, Timing has the lowest Verified scores and credit retention, with weaker support judgments for correct Timing answers. Models with similar Need scores still differ in answer completeness on open questions. Input controls and SFT/GRPO examine context dependence and adaptation. Evaluating answers alongside rationales locates where models fail across understanding, response selection, and justification.