AI summaryⓘ
The authors study how advanced language models that understand both language and video can help users by guiding them in real-time. Because users can achieve goals in many different ways, the authors created a new benchmark called OmniAssistBench, which tests models on following a fixed path based on video information. They built this dataset by breaking down existing online videos into interactive segments, which took a lot of expert effort. Their tests found that current models still make mistakes, especially when understanding visual cues like hand gestures or keeping track of previous steps in a conversation. The authors conclude that these models still need improvement to become reliable video assistants.
omni-modal large language modelsreal-time video assistanceinteractive video understandingbenchmark datasetmulti-turn interactionvisual promptscontext trackinguser goalsvideo segmentationnatural language understanding
Authors
Xianyun Sun, Chaoyou Fu, Zhengye Zhang, Feiyang Duan, Qingyuan Cao, Yonghui Niu, Sihang Yuan, Ge Zhang, Caifeng Shan
Abstract
Recent omni-modal large language models (Omni-LLMs) show great potential as real-time video assistants, which continuously perceive environments and guide users to achieve specific goals. Unlike traditional passive video understanding, interactive assistants should actively combine visual states, user goals, and prior knowledge to provide effective help. Evaluating this is rather challenging, as the model's unpredictable response dynamically changes the user's subsequent actions, which static offline datasets cannot accommodate. To address this bottleneck, we introduce OmniAssistBench. To solve the issue of diverging interaction paths where the same user goal can be achieved through various methods, we provide models with predefined priors derived from the source video, requiring them to guide users along the exact same routes. Since real interaction videos are rare, we construct the dataset by reverse-engineering existing Internet videos. We deduce logical user goals and segment the videos into multi-turn clips to simulate continuous interactions. This rigorous pipeline required over 1000 expert person-hours to build the dataset. Results show that the proprietary Gemini-3-Pro reaches 66.4 out of the max point of 100, while the open-source Qwen3-Omni-Instruct achieves 51.2. Although current models generally understand user inputs, they frequently provide incorrect or incomplete answers. Specifically, they struggle with visual prompts (e.g., hand gestures), fail to maintain historical context during multi-turn interactions, and fail to delay response until the target event. Results indicate substantial room for improvement before models can become reliable assistants.