Watch Think Interact improves answering multiple questions from streaming videos
Watch-Think-Interact: Bootstrapping Long-Horizon Multi-Turn Streaming Video Reasoning with Reinforcement Learning
Artificial Intelligence
Summary
Watching videos and answering questions as they come is hard because you might miss important parts before you know what to focus on. The authors created Watch-Think-Interact (WTI), a system that keeps smart notes linked to video times and can go back to look at key details if needed. WTI decides when it has enough info to answer, keeps watching if it doesn’t, or recalls past video parts to find answers without replaying everything. They trained WTI on lots of timed questions and showed it performs better than other open tools for answering questions about streaming videos.
What this means in practice
- •For video platform developers: Build systems that answer multiple user questions about ongoing videos without needing to replay the entire video history.
- •For smart assistant engineers: Create assistants that provide real-time answers during video streaming by efficiently managing memory and focused recall.
Authors
Ziheng Huang, Yicheng Bao, Xueheng Li, Zhenkun Gao, Bangwei Liu, Kunquan Li, Yuxiang Shen, Bangyan Li, Xuejiao Wang, Changbo Wang, Gaoqi He
Abstract
Streaming video assistance requires models to answer asynchronous questions from an observed prefix under a fixed context budget. Existing approaches model response timing or compress history, but an online state formed before future questions are known can omit visual details before later questions reveal their relevance; the retained state alone cannot recover them. We introduce Watch-Think-Interact (WTI), a closed-loop framework for multi-question streaming video reasoning. WTI maintains compact natural-language memory entries tagged with source-video time ranges; these entries support direct reasoning when sufficient and otherwise anchor selective recall of finer visual evidence. For each question, WTI answers when current context and memory suffice, continues watching when required evidence has not appeared, or recalls a relevant past interval and decides again after incorporating the returned chunks, without replaying the full observed history. To train this behavior, we construct WTI-82K, comprising 82,335 timed questions across 4,812 causally aligned trajectories, and develop Stream-GDPO to optimize complete multi-question streaming rollouts using trajectory-level feedback for response timing, source-video recall, and memory updates. WTI achieves state-of-the-art aggregate performance among the compared open-source streaming baselines, reaching 83.3% on StreamingBench and 73.6% weighted overall accuracy on OVO-Bench.