AutoSkill improves long video question answering by adapting frame selection
One Skill Does Not Fit All: Automatic Discovery and Taxonomy-Guided Routing of Frame-Selection Skills for Long-Video Question Answering
Computer Vision and Pattern Recognition
Summary
Long videos contain a vast amount of information, but answering questions about them quickly is hard because the system can only look at a few important frames. The authors found that different types of questions need different ways to choose these key frames. Their method, AutoSkill, automatically discovers and assigns the best frame-selection strategies for different question categories without needing direct video or answer information from the new videos. This improves the ability of video language models to answer questions on multiple long video datasets.
What this means in practice
- •For video analytics teams: Improve accuracy when answering questions about long videos by automatically selecting the best frame-selection strategy based on question type.
- •For media content indexing services: Provide faster and more relevant video summaries by adaptively choosing frames tailored to different query types without accessing target video content.
Authors
Jian Hu, Zixu Cheng, Da Li, Wei Li, Ziquan Liu, Shaogang Gong
Abstract
Long-Video Question Answering (LVQA) requires locating decisive evidence in hour-scale videos under a limited frame budget. Most training-free methods apply the same frame-selection strategy to all questions, despite substantial variation in the evidence required by different question types. Our analysis shows that the relative effectiveness of frame-selection strategies varies across semantic categories and benchmarks, motivating adaptive evidence acquisition. In this paper, we introduce AutoSkill, a source-supervised framework for automatically discovering and routing executable frame-selection skills. Starting from a small labelled source pool, LLM agents iteratively propose, implement, evaluate, and refine candidate skills. For a target benchmark, AutoSkill uses only unlabelled question and option text to induce a shared semantic taxonomy, rewrite labelled source examples into the target style, and estimate a category-to-skill mapping. Neither target videos nor target answers are used in this process. At inference time, each question is assigned one skill, which selects the frames used in a single inference of the frozen video MLLM. Across five long-video benchmark splits, AutoSkill improves Qwen2.5-VL-7B and Qwen3.5-4B by 2.4% and 1.2%, respectively, demonstrating the effectiveness of our AutoSkill.