Models struggle to use past video history accurately for questions
TwinPast: Benchmarking Historical Evidence Use in Streaming Video Understanding with Evidence Contrastive Video Pairs
Computer Vision and Pattern Recognition
Summary
Many video understanding models can answer questions correctly using only recent video frames or even just the question and answer options, without truly using the past video context. The paper introduces TwinPast, a new test with paired videos where only the historical evidence changes, to check if models actually base answers on that history. Results show that current models often fail to correctly use historical information and sometimes pick facts from the wrong time. The authors highlight the challenge of connecting memory of past events with reasoning based on their timing and relationships.
What this means in practice
- •For video system developers: Improve video AI systems to better use historical video context for answering time-dependent questions by testing on TwinPast benchmark pairs.
- •For multimedia content analysts: Use TwinPast to evaluate and refine tools that analyze event sequences and temporal relations in streaming video data.
Authors
Haotian Yang, Yong Chen, Cheng Peng, Haobo Wang, Zhongle Xie, Xuanru Li, Gang Chen
Abstract
High accuracy on streaming video benchmarks does not establish whether a model uses visual history. Our review of OVO-Bench and StreamingBench finds that recent frames suffice for 32% of sampled historical questions, while models also answer many questions correctly from the question and options alone. We introduce TwinPast to test whether answers correctly follow changes in history, using 350 human-reviewed video pairs spanning five historical tasks. Each pair holds the recent video context, question, and ordered options fixed while changing critical historical evidence and the correct answer. Each video is evaluated independently, and PairAcc requires correct answers for both histories. We evaluate 34 model and input configurations. For Qwen3-VL-8B and DeepSeek-Flash, historical inputs improve performance on TwinPast, reversing the recent-frame advantage on existing benchmarks. The strongest evaluated streaming system reaches 49.14% PairAcc, compared with 66.57-91.43% for the evaluated general-purpose models. This gap persists across all five tasks. Case studies further show that models select states from the wrong time or contradict recorded event order despite retaining the relevant evidence. These findings highlight the need to connect historical memory and retrieval with reasoning that uses evidence according to the question's temporal references and event relations.