Summary
Understanding what people want to do next, especially when they don't say it out loud, is hard for current computer systems that watch videos or screens. The authors created a new test set called StreamVibe-Bench to better measure how well models can guess what users want to do as it changes over time. They also built StreamVibe-VL, a model that watches ongoing actions and tries to respond in a helpful way without waiting for explicit commands. To keep track of how user intent changes, they designed StreamVibe-Harness, which remembers past intentions and adapts responses over longer interactions. Their experiments show these tools make video-based interaction more timely, accurate, and consistent than past methods.
What this means in practice
- •For user interface designers: Create interactive systems that anticipate user actions by tracking subtle behavioral cues over time in video streams.
- •For customer support teams: Deploy video agents that proactively assist users by understanding evolving needs without waiting for explicit requests.$Commercial implications: Enables conversational AI products that better predict user needs in real time, improving support automation.
Authors
Chengzhi Liu, Yuxiang Lai, Yuzhe Yang, Yanqing Liu, Jiheng Hou, Yue Cao, Haonan Wang, Ruixi Fan, Lan Wang, Sheng Liu, Songyou Peng, Yale Song, Kevin Qinghong Lin, Xin Eric Wang, Jui-Hsin Lai, Youbao Tang
Abstract
Vision-language models are shifting from offline passive understanding toward online proactive interaction. However, prevailing methods largely rely on explicit requests while miss the implicit user intent (e.g., "where should I click next?" while a user navigate a website), meanwhile fail to update the state simultaneously as user requests might update continuously. These limitations make them less suited to intent-aware interaction in practical. To address these challenges, we introduce StreamVibe-Bench, a new benchmark serving as testbed for user intent interaction, with 4.1K evaluation samples and 85K training samples spanning explicit, implicit, and dynamic intents. Building on this benchmark, we develop StreamVibe-VL, a streaming proactive model that infers implicit user intents from continuous observations and dynamically tailors its interactions as these intents evolve. We further introduce StreamVibe-Harness, which maintains user intents as persistent states with a fleixble lifecycles via a hierarchical memory to track the latest user intent and environmental state. Our experiments reveal three key findings: (i) StreamVibe-VL preserves strong general video understanding while improving response timing and answer quality by 10.7% over other baselines; (ii) existing proactive models remain limited in understanding user intent and deciding when to respond, especially under implicit and dynamic intents; and (iii) StreamVibe-Harness further improves response consistency and adaptation over sustained interaction, resulting in a 12.3% gain in proactive interaction.