Event grounded self distillation improves real time video understanding
EGSD: Event-Grounded Self-Distillation for Streaming Video Understanding
Computer Vision and Pattern Recognition
Summary
Understanding videos as they happen is tricky because the system needs to remember important details without seeing the whole video at once. The authors found that existing methods that teach themselves using extra information don't work well for live video because the teacher model has more info than the student and focuses too narrowly on parts related to the question. They created a method called Event-Grounded Self-Distillation (EGSD) that helps the system remember important events more effectively and balances attention to all relevant parts. Tests show this method improves memory and accuracy in video understanding tasks.
What this means in practice
- •For video streaming platform developers: Enhance live video content analysis by maintaining more informative and diverse event memories for real-time user interactions.
- •For real-time video analytics engineers: Build systems that incrementally update event-based memory to improve accuracy and recall in continuous video feeds without waiting for future inputs.
Authors
Yuwei Miao, Xuesheng Zhang, Wenhao Zou, Jixia Zhang, Jianwei Lv, Bo Yuan, Junfeng Wang, Shiao Xie
Abstract
Real-time video understanding requires incrementally maintaining a memory of streaming content, and optimizing this requires dense process signals. On-Policy Self-Distillation (OPSD), which lets one model serve as both teacher and student with the teacher receiving additional privileged information such as the question and ground-truth (GT) answer, can supply such token-level signals. However, applying it directly to streaming video raises two problems. (1) The student cannot be optimized end-to-end, where memory is written before the question arrives, yet the teacher scores it with the question-and-GT privilege, misaligning their preferences. (2) Effective-entity memory collapses, where the question-and-GT privilege makes the teacher favor only question-relevant entities, and token-mean averaging over a memory renders its signal invariant to how many entities that memory covers, both driving memory against the streaming need for diversity. To address these issues, we propose Event-Grounded Self-Distillation (EGSD), which characterizes streaming memory as an incremental update over verifiable Events (key visual entities, actions, and details) and targets the two problems on this basis. For problem (1), we adapt the OPSD signal into a multiplicative weight combined with the outcome reward; for problem (2), we re-weight the teacher with Events as privileged information to counter its question-relevance bias, and add an entity-coverage reward to supply the coverage preference the token-mean teacher lacks. Extensive experiments on mainstream online and offline benchmarks show EGSD achieves strong performance, reaching 79.8% on StreamingBench and 73.4% on the OVO-Bench Real-Time track, while memory analysis shows effective-entity recall rises 17.4% at only 6.8% more memory length.