Video models better track time by boosting fading signals
Before It Fades: Reinforcing Temporal Representations at Inference Time in VideoLLMs
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Video language models often struggle to understand how actions unfold over time because key information fades before their final decision. The authors found that this fading happens in the middle of the model, where it learns about the order of frames but then loses it toward the end. They developed a technique called Temporal Activation Injection to capture and refresh these fading signals during the model’s prediction process. This approach doesn’t require extra training and helps models reason about time more accurately without hurting other video tasks.
What this means in practice
- •For video ai developers: Improve video language model accuracy by enhancing temporal understanding during inference without retraining.
- •For media content analysis teams: Enhance tools that analyze video sequences by better capturing the order and progression of visual events.
Authors
Youngwoo Shin, Yusung Ro, Minseo Kim, Junmo Kim
Abstract
Video Large Language Models (VideoLLMs) receive frames in sequential order and interpret how visual content evolves along the temporal axis, yet temporal reasoning remains a persistent weakness across architectures. Reversing the frame order of a video, a transformation that should invert temporal answers, often leaves the final prediction unchanged. We investigate where this failure originates by defining the temporal divergence vector $τ_l$, the layer-wise representational difference induced by reversing temporal order. Tracking its magnitude across layers reveals a consistent temporal divergence profile where the divergence peaks at intermediate layers and progressively diminishes toward the output. We confirm this peak is specific to temporal reasoning and functionally critical for predictions, establishing that VideoLLMs acquire temporal information at intermediate layers but fail to maintain it to the output. This progressive fading motivates our method, Temporal Activation Injection (TAI), which extracts $τ_l$ at the peak of the profile for each input and reinjects it into subsequent layers following the measured decay. TAI requires no training and consistently improves temporal reasoning across three VideoLLMs and four benchmarks with negligible impact on non-temporal tasks. Code is available at https://github.com/Youngwoo-git/Before-It-Fades.