Adaptive keyframe selection improves long video question answering accuracy
FORTE: Adaptive Scoring and Exact Keyframe Selection for Long-Video Question Answering
Computer Vision and Pattern Recognition
Summary
Answering questions about long videos is hard because computers can only look at a few important frames. The authors designed a method called FORTE that picks these key frames smartly by quickly guessing which parts of the video are relevant and balancing attention across the whole video. This approach helps find the best frames to answer questions without missing important moments. Their tests show FORTE picks better frames and leads to more accurate answers than other methods.
What this means in practice
- •For video analytics teams: Improve question answering accuracy by selecting relevant frames adaptively in long video analysis pipelines using limited computational resources.
- •For multimodal ai developers: Integrate FORTE’s scoring and selection method to efficiently feed relevant frames to large language models for better understanding of lengthy videos.
Authors
Haifeng Huang, Biyin Xu, Chunsheng Xin, Yang Li
Abstract
Query-aware keyframe selection enables multimodal large language models (MLLMs) to process long videos using only a small set of question-relevant frames. Existing score-based methods, however, typically search within a fixed, uniformly sampled candidate pool, preventing evidence outside this pool from ever being selected. Given a limited relevance-scoring budget, the key challenge is to allocate evaluations adaptively to promising frames while continuing to explore underrepresented temporal regions. We introduce FORTE, a training-free framework that addresses this challenge through two stages: adaptive relevance scoring and global keyframe optimization. Starting from sparse, uniformly distributed observations, our efficient Gaussian-process relevance predictor estimates relevance for unscored frames, exploiting temporal locality and the approximately banded kernel structure to reduce the core computation from cubic to linear time in the number of frames for fixed bandwidth. The scoring stage then selects which frames to score next by balancing predicted relevance with temporal coverage, prioritizing promising regions while also exploring less-represented parts of the video. The optimization stage selects the final keyframes by maximizing an objective that jointly captures measured relevance and temporal coverage. We derive an exact algorithm that leverages the logarithmic coverage structure to identify the optimal subset of the scored candidate pool in time linear in the pool size, for a fixed final-frame budget. Experiments on four long-video question-answering benchmarks show that FORTE achieves the highest observed mean accuracy among the compared selectors under every tested scoring budget. Further evaluations demonstrate its consistent effectiveness across different relevance scorers and downstream MLLMs.