Efficient frame selection improves long video understanding with bayesian learning
From Label Priors to Task Evidence: Efficient Long-Video Frame Selection via Bayesian GRPO
Computer Vision and Pattern Recognition
Summary
Long videos are hard for AI to understand because they require a lot of effort to process every frame. The authors found that existing methods pick important frames based on fixed labels that don’t always match the needs of the AI models making sense of the video. They created a new way that lets a small program learn which frames are most useful by getting feedback from a larger AI model’s answers. This method works well across different AI systems, saves computing power, and doesn’t need retraining for new tasks.
What this means in practice
- •For video platform engineers: Reduce computing costs by selecting key frames efficiently for AI-based long video content analysis without retraining per AI model.
- •For ai product developers: Integrate lightweight frame selectors that improve long-video question answering by aligning selection with downstream AI task needs.$Commercial implications: Enables selling advanced video analysis tools that work efficiently across multiple AI models with no retraining required.
Authors
Shuochen Chang, Bingjie Gao, Qingyang Liu, Qianli Ma, Yibo Miao, Haonan Zhao, Zhaohe Liao, Xiaofeng Zhang, Jiangtong Li, Li Niu
Abstract
Multimodal Large Language Models (MLLMs) face significant scalability challenges in long-video understanding due to quadratic attention complexity and context window constraints. Existing retrieval-based approaches typically rely on supervised learning from static keyframe annotations. However, these label priors often fail to align with the actual information required by downstream MLLMs for complex reasoning, creating a misalignment between the training target and inference needs. To bridge this gap, we propose a probabilistic framework that reformulates frame selection as latent-variable inference. We first initialize a lightweight policy using supervised learning to capture general visual saliency. Then, we treat a frozen MLLM as a task environment and use its answer likelihood as dense task evidence to refine the policy via Bayesian Group Relative Policy Optimization (GRPO). Within this formulation, the negative log-likelihood (NLL) over ground-truth answer tokens provides a principled dense reward, while a prior-matching KL term preserves semantic consistency with the elicited label prior. This process transitions the selection principle from static label priors to task evidence. Extensive experiments on Video-MME, MLVU, and LongVideoBench demonstrate that our selector achieves competitive accuracy with MLLM-based scorers while retaining the inference efficiency of lightweight methods. The selector is trained once with a frozen MLLM environment and transfers directly to multiple downstream MLLMs without retraining. Code is available at \url{github.com/Miraclecsc/Frame_Selection_GRPO}.