Cuekfs picks key video frames guided by dynamic question clues
CueKFS: Agentic Cue-Driven Keyframe Selection for Long Video Understanding
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
Summary
Understanding long videos by picking just a few key frames is tricky, especially when questions involve multiple parts or hidden details. The authors propose CueKFS, a method that breaks down questions into smaller clues and uses them to search the video actively and repeatedly. This approach helps find more relevant frames than previous methods, improving accuracy while using fewer searches. Tests show CueKFS works better on several video question-answering tasks compared to earlier techniques.
What this means in practice
- •For video platform developers: Improve video search by selecting frames that better answer user questions in long videos, enabling quicker and more precise video browsing.
- •For digital content editors: Create more relevant video thumbnails and summaries focused on viewer queries without manual frame selection, aiding content presentation and review.
Authors
Weitai Kang, Hanieh Deilamsalehy, Yumo Xu, Dewang Sultania, Serdar Cellat, Yan Yan
Abstract
Keyframe selection (KFS) has long produced compact video summaries for browsing and retrieval, and representative frames for thumbnails. More recently, when conditioned on a question, KFS provides an alternative to uniform sampling for long-video question answering by selecting frames that are more relevant to the question. Most methods rank frames by similarity to the question. Yet a relevant frame may score poorly when the question combines subjects or moments that no single frame shows, or requires implicit information absent from its wording. Other methods try to break down the question into subqueries, but suffer from inaccurate decomposition due to their static initial context. Therefore, we propose CueKFS, a training-free method that reformulates question--frame matching as comparing frames against a set of dynamically generated visual cues. From an initial set of salient frames, we decompose the question into cues. Each cue concurrently probes the video to navigate to its own evidence. A reasoning VLM then agentically revises the cue set against its evidence to re-explore the video. CueKFS then allocates the budget across the surviving cues. Across three benchmarks, CueKFS establishes state-of-the-art results in all 27 evaluated settings with available prior results, achieving budget-averaged gains of up to +4.54% over the previous baseline and a median of only two VLM calls. We further provide a detailed behavioral analysis of CueKFS, showing that agentic cue refinement drives active re-exploration of the video, yielding relative similarity gains of up to 92% over the initial context.