Proactive video assistants need better timing and response skills
From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios
Machine Learning
Summary
Video assistants are tools that watch videos and help by understanding what’s happening. This paper shows that current video assistants can recognize daily activities but struggle to predict risks and decide when to help. The authors created a new test to judge these abilities and built a better assistant using extra training data. Their work highlights that helping well is not just about seeing clearly but also about choosing the right moment and knowing when to hold back.
What this means in practice
- •For home automation developers: Design video assistants that not only recognize events but also predict risks and provide timely help in domestic environments.
- •For surveillance system designers: Create systems that analyze live video to anticipate potential safety issues and decide when to alert or intervene.
Authors
Boyu Cui, Junhao Luo, Zhihao Wang, Linghan Chen, Zejun Li, Jiwen Zhang, Siyuan Wang, Zhongyu Wei
Abstract
Recent multimodal large language models have advanced video understanding, but existing benchmarks mainly evaluate whether models can interpret observed visual content. In real-world scenarios, assistants must additionally anticipate what assistance is needed and decide when to provide it. To address this gap, we introduce \textsc{P$^3$IBench}, a benchmark for proactive video understanding in real-life streaming scenarios. It evaluates three complementary capabilities: daily perception, risk prediction and intervention. We benchmark 12 representative open-source and proprietary video-language models, including streaming and adapted non-streaming systems. Our evaluation shows that current models struggle to make well-calibrated response decisions. Finally, we construct additional proactive training data and train \textsc{P$^3$I-Assistant-4B}, achieving improvements on several tasks, though gains remain uneven across capabilities. Together, these findings show that effective proactive assistance requires not only visual understanding, but also precise response timing and appropriate restraint.