Video models improve event recall by separating views and supervision
Supervising the Visible State: Separating Representation from View Selection in Streaming Video
Computer Vision and Pattern Recognition
Summary
When teaching computers to understand actions in videos, it’s tricky because the same action can look different depending on how often or in what order it appears. The authors of this paper made a new approach that teaches the computer not just to recognize actions, but also to understand details like the order and changes in events, without getting confused. They also let the system choose what parts of the video to look at based on the situation, instead of always looking at everything. These ideas together lead to better accuracy in understanding videos while using less computing power.
What this means in practice
- •For video analytics developers: Build video systems that better identify and remember actions with fewer frames processed by separating what the model learns from what video views it uses.
- •For autonomous vehicle engineers: Enable real-time understanding of events around a vehicle by adopting adaptive view selection guided by supervision that helps distinguish event order and relations.
Authors
Yihao Hu, Pan Wang, Yitong Lin, Zhihao Wen, Deyuan Liu, Hang Wang, Mingda Wan, Yiwen Zhao, Yutong Li, Yuchen Mao, Tao Lin
Abstract
An action seen once and the same action seen three times receive the same action-recognition label, so that supervision does not require a model to count. Answers shared across histories that differ in event order, accumulation, or relations leave those distinctions unconstrained; we call this state under-specification. Visible-State Supervision (ViSS) addresses it by training on source-grounded probes of event identity, order, accumulation, relation change, and answer sufficiency; under the same recent-frame interface it improves OVO-Bench over a strict-matched generic temporal supervision baseline while remaining comparable on StreamingBench, with the largest gain in hallucination detection. Test-Time Causal View Search (TVS) then adapts evidence access while keeping the learned model fixed, expanding from recent frames to sparse event and compact history views on demand. The two interventions separate representation from view selection: supervision particularly benefits hallucination detection and causal reasoning, access particularly benefits event recall and forward reasoning. Together they reach the best reported accuracy among the published systems compared here on StreamingBench and OVO-Bench at a fraction of the processed frames of exhaustive selection, and strict replay shows that most of the remaining gap to a three-view oracle is a correct answer that was later replaced. Streaming accuracy depends on what supervision asks the model to keep, what evidence inference exposes, and whether the model keeps a correct answer once it has one.