Streaming vision language models adapt future perception without retraining
Foresight: planning future perception in streaming VLMs without retraining
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Vision-language models usually process video streams with a fixed way of understanding what they see, missing chances to adjust their focus based on what might happen next. The authors show these models can actually predict near-future events and use this to change how they work, without needing to be retrained. They build a system called FORESIGHT that runs two linked language models—one keeps up with the video, while the other looks ahead to plan how to process upcoming information. This method helps the model perform better on tasks involving streaming video, especially when important details show up later.
What this means in practice
- •For video analytics developers: Create video processing systems that adjust their focus dynamically based on predicted future events for improved real-time understanding.
- •For security monitoring teams: Deploy streaming vision-language tools that anticipate important activities and adapt analysis strategies without retraining to enhance event detection.
Authors
Ashok Prasad Neupane, Dipan Bartaula, Ankit Belbase, Saugat Adhikari, Samip Ghimire, Saroj Poudel, Binod Bhattarai, Danda Pani Paudel
Abstract
Existing streaming vision-language models (VLMs) continuously perceive and reason over visual streams, but their computational pathways remain fixed throughout inference. Consequently, they cannot adapt computation to evolving scene dynamics, where different future events demand different levels and forms of perception. We show that streaming VLMs inherently possess the ability to anticipate the immediate future, and leverage this capability to dynamically configure future computation in a training-free manner. Realizing such anticipatory computation, however, is very challenging: future anticipation must be sufficiently reliable to guide computation, planning must run concurrently with streaming inference, and online reconfiguration must incur negligible overhead. To address these challenges, we introduce FORESIGHT, a dual-stream architecture comprising two Siamese LLMs with shared weights, input encoders, and KV cache. The first LLM continuously processes incoming tokens, while the second runs ahead of the stream to anticipate future context, plan future computation, and generate task responses without interrupting streaming inference. Each plan decides when to reason next, what to check then, and how densely to sample, keeping transient evidence separate from persistent control. The resulting computation plan is executed online through an efficient reconfiguration protocol with schemaguided decoding and lightweight diff-based updates, enabling dynamic adaptation with low overhead. With a frozen Qwen3-VL-8B backbone, FORESIGHT achieves 23.0 mean joint F1 on OmniPro Online evaluation beating strongest trained baseline by 9.5%, while improving the backbone by 6.7 on StreamingBench and 15.4 on OVO-Bench, with the largest gain of 18.7 when evidence arrives later in the video stream. Our source code will be made publicly available.