Multimodal language models analyzed for real-time social understanding gaps
GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?
Artificial Intelligence
Summary
People understand the social meaning of conversations by picking up hints as they happen, like facial expressions and tone changes. The authors created GesBench, a new test to see when and where AI models notice these social cues during conversations. They found that humans make quick and changing guesses about speaker intentions, but current AI models often delay or repeat guesses. GesBench helps shift attention from whether AI finally understands to when and where it does during interactions.
What this means in practice
- •For multimodal ai developers: Evaluate and improve when multimodal language models detect social cues during conversations to enhance real-time interactions.
- •For human-computer interaction designers: Design dialogue systems that better align with human social perception by understanding timing and focus of social signal processing.
Authors
Zonghuan Li, Hayley Hung, Chirag Raman
Abstract
Human social perception is incremental: we infer intent before an utterance is complete, revise our interpretation as new cues arrive, and attend to specific facial, gestural, or prosodic signals that shape social meaning. Current multimodal LLM benchmarks largely evaluate final-answer understanding over complete episodes, rewarding post-hoc recognition of social meaning rather than the unfolding perception required for real-time interaction. Drawing on interactionally embedded Gestalt principles of multimodal human communication, we introduce GesBench: a human-grounded benchmark for cue-level social perception in multimodal LLMs. GesBench uses cumulative sub-utterance clips to measure when models form stable interpretations of speaker intention and social affordance, and human gaze-guided corruption to evaluate whether regions attended by humans are functionally important for model judgments. Built from more than 6000 utterance-level clips from scripted and unscripted conversational datasets, GesBench includes human annotations of unfolding judgments and gaze on a representative subset. Across open and closed multimodal models, we find a clear human-model gap: humans form predictive understandings early and revise more often, while models often hedge or repeat across different clips. GesBench shifts social evaluation from whether models eventually recognize meaning to when and where that meaning is formed. Our codes are publicly available at https://anonymous.4open.science/status/GesBench-share-212C.