Syncpro improves timing and accuracy in proactive video understanding
SyncPro: Synchronous Reasoning and Responding for Proactive Streaming Video Understanding
Computer Vision and Pattern RecognitionComputation and LanguageMachine Learning
Summary
People use video assistants that watch live streams and respond when something they want happens. The challenge is getting these assistants to keep track of new information as it happens and reply exactly when needed without delay. The authors created SyncPro, a system that manages two separate but connected tasks: watching the video closely and responding quickly to user requests at the right time. Their approach lets the system update its understanding and reply at the same time, making answers more accurate and timely. They trained SyncPro on a large dataset and tested it on multiple challenges, showing better performance than earlier methods.
What this means in practice
- •For streaming video platform developers: Improve real-time video assistants to respond immediately when requested events happen in live streams.
- •For interactive media producers: Create proactive video systems that deliver timely and precise answers during live event broadcasts.
Authors
Xueyun Tian, Bingbing Xu, Jiaming Liu, Jinpeng Yu, Yang Shi, Yu Zhao, Ruihua Huang, Yuanzhuo Wang, Huawei Shen
Abstract
Proactive streaming video understanding requires assistants to track unfolding events and respond when a user's requested condition is met. To make these decisions, assistants must accumulate relevant evidence and reassess response conditions as new observations arrive. Existing work offers complementary capabilities for this process: proactive video assistants learn when to respond, while streaming video reasoning methods generate scheduled intermediate text to organize observations and support subsequent answers. Together, these capabilities provide a foundation for proactive reasoning, but how to coordinate ongoing reasoning with timely responses remains underexplored. In a direct adaptation, fixed reasoning intervals can trigger updates when nothing relevant has changed or postpone useful updates until the next scheduled step. Sharing one generation path also couples two distinct requirements: preserving evidence for future decisions and producing only the response requested by the user. Moreover, responses generated after reasoning must wait for it to finish, even when sufficient evidence is already available. To address these challenges, we introduce SyncPro, a synchronous dual-stream framework with specialized generation paths for evidence tracking and user-facing responses. Its Inner and Response branches share causal history and activate independently, allowing relevant evidence changes to trigger internal updates and responses to proceed concurrently with ongoing reasoning. We construct 189K dual-stream examples and train SyncPro through supervised fine-tuning followed by trajectory-level reinforcement learning. Experiments across three proactive benchmarks show improved response timing and more accurate answers to user requests.