BLIP-Live improves timely and accurate responses in video streaming AI
BLIP-Live: Balancing Timeliness & Firing Precision in Proactive Streaming Video Agents
Computer Vision and Pattern Recognition
Summary
Interactive video systems need to respond quickly but avoid giving wrong or repeated answers. The authors created BLIP-Live, a framework that helps AI decide when to respond based on how much useful information it has seen. They built a large dataset with human feedback on good response timing and content, and invented a new way to measure how well the AI balances quickness with accuracy. Using these tools, they improved existing vision-language AI to give better and faster answers during ongoing video streams.
What this means in practice
- •For interactive system developers: Create video agents that respond promptly and correctly during live interactions using the BLIP-Live framework and dataset.
- •For customer service platform teams: Integrate improved proactive video understanding agents to enhance response quality and timing in video-based customer support.$Commercial implications: Enables advanced streaming video AI that delivers faster and more accurate customer responses, improving platform service capabilities.
Authors
Kanchana Ranasinghe, Honglu Zhou, Dominick Reilly, Le Xue, Shusheng Yang, Juan Carlos Niebles, Silvio Savarese, Ran Xu
Abstract
Streaming video understanding is becoming increasingly important for real-world interactive systems. Such systems must be proactive for natural and useful human interaction: they should respond as soon as sufficient evidence becomes available while avoiding premature, redundant, or incorrect responses. We present BLIP-Live, a framework unifying data, metrics, harness, and training around a shared formulation of proactivity as effective causal response decisions balancing the two competing objectives of responding vs silence. First, we construct BLIP-Live-780K dataset containing human annotations of proactive response content and timing, which defines this desired behavior. We then propose PAUC-F, a metric to effectively measure our desired proactive behavior. PAUC-F combines existing response timeliness metrics with firing precision to reward early useful responses without rewarding excessive firing. With our desired behavior defined and measurable, we finally investigate model-agnostic inference harnesses that draw out proactive streaming capabilities of offline vision language models (VLMs), together with training pipelines that learn to further improve their proactive responsiveness. Across both settings, we find that combinations of sliding visual windows, online interaction memory, and structured response outputs consistently yield strongest proactive performance. Our experiments result in two models, with our BLIP-Live-Qwen model achieving state-of-the-art performance across four proactive streaming benchmarks, with gains up to 44% over baselines and outperforming multiple closed API models. Our code and data will be released publicly.