EARS uses sound and motion to pick key video frames for analysis
EARS: Evidence-Aware Retrieval via Sound and Motion for Budgeted Video Reasoning
Computer Vision and Pattern Recognition
Summary
Watching every part of a video can miss brief but important moments if you only look at a few frames. The authors propose EARS, a simple way to pick the most interesting frames by looking at movement and sound instead of just regularly spaced frames. This method improves how well a video language model can find important moments in videos with noticeable action or noise, but it is less effective for quiet or dense videos. EARS works without extra training or searching, making it a lightweight way to select frames for video understanding.
What this means in practice
- •For video surveillance teams: Improve locating brief events in surveillance videos by selecting frames with motion and sound to analyze efficiently.
- •For security monitoring operators: Provide faster review of sparse-event security footage by automatically focusing on frames with notable motion or audio activity.
Authors
Shih-Yao Lin
Abstract
Uniform frame sampling can miss brief evidence in sparse-event videos when the VLM sees only K frames. We propose EARS (Evidence-Aware Retrieval via Sound and motion), a training-free, query-free front-end that ranks frames using optical-flow motion and audio energy, then applies temporal non-maximum suppression for diversity. The evidence score alone has frame-level AP 0.39; used only to select frames for the VLM, EARS with temporal diversity reaches localization AP 0.94, compared with 0.88 under Uniform sampling. In the updated matched study, at K = 1 EARS reaches XD-subset F1 0.660 versus 0.614 for centered Uniform (the paired interval includes zero); earlier backbone studies use separate protocols. On mostly-silent UCF-Crime, however, EARS slightly trails Uniform. A single configuration is used across both benchmarks; the new controls do not repeat the original held-out search. EARS therefore targets low-budget videos whose salient evidence aligns with motion or audio; on dense narratives or stealthy events, uniform or semantic selection remains preferable, and cheap modality statistics cannot reliably route between them. Code: https://github.com/mikeslin-ir/EARS