Phase-aware video evidence selection improves event understanding accuracy
dKFD: Phase-Structured Evidence Allocation for Fixed-Budget Localized Event Understanding
Computer Vision and Pattern Recognition
Summary
Understanding events in videos often means picking a few important frames from the whole clip, but picking all frames from a single pool can miss the story before and after the main action. The authors show that splitting the video into phases—before, during, and after the event—and selecting frames from each phase leads to better detection of events. Their method, called dKFD, picks frames more reliably and improves both recognizing the event and pinpointing when it happens. This approach works best for videos centered on specific events rather than for general video summaries.
What this means in practice
- •For video analysis teams: Identify key frames from different event phases to improve anomaly detection and temporal localization in surveillance footage.
- •For autonomous vehicle engineers: Enhance understanding of traffic events by selecting phase-related video frames, aiding in accident recognition and response.
Authors
Aditya Bagri, Ashutosh Kumar, Chaitanya Lakhchaura, Avinash Anand, Zhengkui Wang, Rajiv Ratn Shah
Abstract
Sparse video understanding often requires selecting a small set of visual evidence under a fixed frame budget. Most sparse selectors allocate this budget globally, allowing all frames to compete with one another. For temporally localized events, this can be a poor inductive bias: useful evidence is often distributed across pre-event context, the event itself, and post-event consequences. We study fixed-budget evidence allocation for localized event videos and show that globally competitive Top-$K$ selectors can preserve recognition and grounding while producing unstable event evidence. On DoTA Video Anomaly Recognition, Global Top-$K$ obtains competitive recognition and temporal grounding, but low selector-event alignment at $K=12$ (Frame AUC $51.5 \pm 10.8$). We propose dKFD, a phase-structured differentiable selector that reserves evidence capacity across pre-event, event, and post-event phases after full-sequence temporal encoding. Under matched-budget multi-seed evaluation, dKFD improves Frame AUC by $+30.97$ over a matched Global Top-$K$ selector at $K=12$ ($p<0.01$), while yielding modest but statistically significant recognition gains and comparable temporal grounding. Mechanism ablations show that phase supervision is load-bearing: removing it reduces Frame AUC to $41.1 \pm 12.0$ even when phase-partitioned budgets are retained. Downstream diagnostics on VRU-Accident show consistent gains over learned Global Top-$K$ across VLM families, while dense captioning reveals a boundary condition where uniform sampling remains competitive. These results support phase-structured allocation as a controlled fixed-budget approach for event-centric sparse evidence selection, not as a universal video summarization strategy.