Trace benchmarks streaming video models by timing and response quality
TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding
Computation and LanguageComputer Vision and Pattern Recognition
Summary
Understanding video as it streams is different from just getting one score at the end. The authors created TRACE, a way to check not only if video questions are answered right but also when answers come, how the video history is tracked, and how the models decide to respond. They tested popular models and found that similar scores can hide big differences like delays or wrong timing. This means to judge video AI better, we need to look at how they behave over time, not just their final accuracy.
What this means in practice
- •For video software developers: Evaluate and compare streaming video understanding systems by measuring not only accuracy but also response timing and workload behavior.
- •For real-time monitoring teams: Monitor streaming video AI systems to better detect false alarms and missed response windows for improved operational reliability.
Authors
Yibo Ma, Qianqian Zhang, Peng Liu, Tiancheng Zhao
Abstract
Streaming video understanding requires models to interpret evidence as it arrives, yet current evaluations often report task scores without specifying when evidence becomes valid, how visual history is maintained, or how responses are triggered. As a result, similar scores may correspond to different workloads, failure modes, and operational behavior. We introduce TRACE (Temporal Audit and Condition-aware Evaluation), a condition-aware benchmark and evaluation framework that makes these factors explicit. TRACE combines temporally audited visual tasks with evidence timing and instruction-dependent trigger annotations, a unified causal Core--Adapter protocol that controls information availability while recording actual history processing and response events, and multidimensional reporting of answer quality, timeliness, response-selection behavior, workload, completion, and reliability. On 1,240 records from 517 videos, we evaluate eight publicly available models or systems in eight configurations. We find that nearly identical QA accuracy can mask substantial differences in completion, answer validity, and generation workload, while proactive performance separates into response quality, response delay, false alarms (responses emitted while no target window is currently valid and a later one remains), and missed target windows. These results show that streaming-video performance should be interpreted as execution-conditioned system behavior rather than a single score. Our benchmark and code can be accessed at \href{https://github.com/om-ai-lab/trace-bench}{https://github.com/om-ai-lab/trace-bench}.