Omni large language models struggle with timely and proactive help
Omni-InteractScope: Benchmarking Proactive and Agentic Interaction in Continuous Omnimodal Streams
Artificial IntelligenceComputer Vision and Pattern RecognitionHuman-Computer Interaction
Summary
Modern AI models that watch and listen continuously to videos and sounds are expected to help users with tasks as events happen. The authors created a new benchmark called Omni-InteractScope that measures how well these models act on events when needed, like starting tools or giving reminders without being asked. They found that current models often respond too late, struggle with unprompted reminders, and rarely provide full step-by-step help. This shows there is still much work to do before such AI assistants work reliably in real-time situations.
What this means in practice
- •For ai assistant developers: Evaluate and improve AI models for timely, proactive tool use and reminders in continuous audio-visual monitoring environments.
- •For human-computer interaction teams: Design multi-modal AI systems that better track user needs and deliver continuous, context-aware responses in real time.
Authors
Yinsong Yan, Ling Wang, Yuxuan Wang, Yifan Yang, Ruiyang Xu, Junming Lin, Qi Chen, Meng Gao, Minghao Han, Yunfei Chu, Qize Yang, Xize Cheng, Jin Xu, Jibin Wu
Abstract
Omnimodal large language models (Omni-LLMs) are increasingly expected to assist users while observing continuous audiovisual streams. However, existing evaluations provide limited insight into how reliably these models connect stream evidence and user needs to timely, sustained interventions. We introduce \textbf{Omni-InteractScope}, a benchmark of 1,479 human-annotated, cross-reviewed interactions with 1,790 temporal targets across eight task groups. It links input modalities, event triggers, intervention types, and user needs, covering event-timed tool invocation and query-free assistance alongside notifications, warnings, state tracking, and continuous responses. Our experimental results on the proposed benchmark reveal three critical limitations of current Omni-LLMs: (1) \textbf{Timely tool invocation remains unreliable}: all 12 evaluated models produce an on-time first response for at most 40\% of tool-invocation targets, and joint timing-and-tool accuracy reaches at most 32.0\%. (2) \textbf{Query-free reminders remain poorly covered}: all evaluated models achieve lower timely reference-target recall on query-free reminders than on explicit questions, measured on their respective target sets. Reminder recall never exceeds 23.0\%. (3) \textbf{End-to-end guidance coverage remains limited}: nine models cover none of the 26 guidance sequences in full, and the remaining three cover only one to three sequences each. Targeted controls further show that tool-invocation scores alone do not establish event grounding and that reminder coverage depends on response history and frequency. These findings motivate developing and evaluating streaming assistants for timely action, initiative under general user needs, and coverage of entire response sequences.