Streaming video models struggle to respond at right moments
ProactiveBench: Can Streaming Video Models Really Interact Like Humans?
Machine Learning
Summary
Understanding what's happening in continuous video streams is hard for computers, especially when they need to decide when to respond like humans do. The authors created ProactiveBench, a way to test if video models can recognize events and respond promptly without being told exactly when to reply. They found many models respond too early rather than missing events, showing these systems still have trouble timing their answers properly. This means current models are not yet good at interacting naturally in real-time scenarios.
What this means in practice
- •For video analytics developers: Improve real-time event recognition systems by benchmarking response timing with ProactiveBench.
- •For interactive ai product teams: Enhance user interaction in live video applications by using models tested for timely and accurate event responses.
Authors
Kaixuan Du, Xin Wan, YuKun Wang, Hang Zhang, Meng Cao, Dai Guan, Ming Chen, Ni Li
Abstract
Streaming video understanding requires models to process continuous multimodal input while maintaining temporal context. Existing evaluations are predominantly reactive: they query a model at a selected timestamp and therefore do not assess when it should respond. Proactive interaction instead requires monitoring a standing request, responding within an appropriate interval after the target event, and otherwise remaining silent. We introduce ProactiveBench, which evaluates models at one-second stream intervals without an explicit response cue. Its six subtasks vary trigger ambiguity and timing tolerance. Event Sensitivity geometrically combines response and silence rates on the same recording; four window-based subtasks distinguish early, in-window, and missed responses; and Duplicate Counting penalizes omissions and repetitions. Premature responses outnumber missed responses for four of the six evaluated systems, revealing a substantial gap in the temporal decision-making required for human-like interaction.