Multimodal models struggle to understand user demands in interactions
Omni Demand Understanding: A Benchmark for Contextual User-Intent Inference in Multimodal Interaction
Computation and LanguageComputer Vision and Pattern RecognitionMultimediaSound
Summary
People often talk and use gestures or visual cues when asking for help from AI assistants, but these requests can be unclear or noisy. The paper’s authors introduce a new test called Omni Demand Understanding to see if AI systems can figure out what users really want from mixed speech, visuals, and conversation history. They found that even some of the best AI models miss a lot of important details and often think someone is asking for help when they are not. This shows that current AI assistants have trouble truly understanding what people want during natural, everyday conversations.
What this means in practice
- •For ai product teams: Improve AI assistants’ ability to detect genuine user demands from mixed speech and visual signals to reduce false activations and enhance user experience.$Commercial implications: This paper enables building smarter AI assistants that better understand users' intentions, making voice and vision multimodal products more reliable and user-friendly.
- •For multimodal system engineers: Evaluate and benchmark multimodal models on their capability to infer user intent across speech, vision, and conversation context to guide development.
Authors
Qi Chen, Yunfei Chu, Haolin He, Yifan Yang, Zihan Liu, Yuxuan Wang, Ziyang Ma, Ruiyang Xu, Meng Gao, Yinsong Yan, Ling Wang, Hui Wang, Wen Huang, Yiheng Chen, Guanrou Yang, Qiuqiang Kong, Jin Xu, Xie Chen
Abstract
Natural audio-visual interaction is emerging as an important interface for AI assistants, allowing users to communicate through speech and vision rather than carefully composed text prompts. However, existing benchmarks of interactive capabilities still focus primarily on response quality, leaving a more fundamental question underexplored: can a model correctly infer the user's underlying demand from complex multimodal interaction? Real-world user demands are often underspecified in speech and must be inferred from multimodal cues and dialogue history. This inference is further complicated by ambiguous or disfluent expression and noisy acoustic environments. Conversely, request-like speech may not constitute a demand to the assistant, leading to false triggers. We establish Omni Demand Understanding (ODU) as a distinct multimodal contextual inference problem: given an interaction stream, a model must detect whether a user demand is present and infer intent from multimodal and conversational context. ODU evaluates this capability along five dimensions, covering both single-turn and multi-turn interactions. We construct ODU-Bench using a challenge-driven taxonomy, taxonomy-guided agentic video generation, and human-recorded interactions, followed by media-grounded annotation and human verification. We evaluate 14 native MLLMs. Even the strongest, Gemini 3.1 Pro, recovers only 44.7% of key information that must be inferred from visual, acoustic, or conversational context. Moreover, 11 of the 14 models exhibit false-trigger rates above 50% on non-demand scenarios. These results reveal a systematic capability gap in current MLLMs' ability to infer contextual user demands. We hope ODU can establish the evaluation of a previously underexplored yet essential capability in multimodal interaction: correctly understanding user demands before generating an appropriate response.