Event chain method improves understanding of long videos over frames
EC-RAG: Event Chain Retrieval-Augmented Generation for Long Video Understanding
Computer Vision and Pattern Recognition
Summary
Long videos are hard for current AI systems to understand because frames are treated separately, missing the story that connects events over time. The authors propose a new way called EC-RAG that breaks a video into meaningful events and links them in order to capture how actions unfold. When asked a question about the video, EC-RAG finds the important events and combines speech, text, and visuals from those events to give better answers. Their approach works well on tests and does not require extra training, making it easier to use with existing video-language models.
What this means in practice
- •For video content analysis teams: Improve answering questions about long video recordings by reasoning on linked events instead of isolated frames.
- •For media editing software developers: Enhance tools that support searching and summarizing long videos based on event chains to provide context-aware navigation.$Commercial implications: Enables commercial video editing products to offer smarter event-based retrieval and summarization features.
Authors
Yuhao Qin, Junbo Wang, Yuke Li, Yining Zhu
Abstract
Current large video-language models (LVLMs) still face challenges when dealing with long videos, mainly because frames are often processed independently, making it difficult to capture temporal dependencies across events. Although retrieval-augmented approaches have been introduced to provide additional context, most of them operate at the frame or snippet level, which limits their ability to model how events evolve over time and relate to each other. In this paper, we propose Event Chain Retrieval-Augmented Generation (EC-RAG), a training-free framework that organizes video content into an explicit event chain before question answering. Instead of retrieving isolated frames or text segments, EC-RAG first partitions the video into semantically coherent segments, represents each segment using multi-modal signals, and then links them into a structured chain that preserves temporal order and captures inter-event relationships. Given a query, the system identifies relevant events within this chain and gathers supporting evidence from the associated modalities. Our approach offers several practical advantages: (i) event-level abstraction that better reflects how video content is naturally structured, enabling more reliable localization compared to frame-level retrieval; (ii) structured multi-modal fusion that aggregates speech, text, and visual cues at the event level, allowing complementary information to be more effectively utilized during reasoning; and (iii) plug-and-play compatibility with existing LVLM backbones, requiring no additional training or reliance on proprietary models. Experiments on Video-MME, MLVU, and LongVideoBench show that this event-centric design consistently outperforms frame-level retrieval baselines, highlighting the importance of modeling temporal structure for long-video understanding.