QSVideo: Query-Conditioned Semantic Temporal Retrieval for Video Understanding
2026-07-06 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors found that video-language models get worse at understanding videos as they get longer, because unrelated parts confuse them. To fix this, they created QSVideo, a system that better finds important parts in videos by focusing on relevance, variety, and timing. Their method turns questions into clearer searches, picks diverse key frames, and aligns important moments over time. Tests show their approach works well on long and live video streams even with limited frames.
vision-language modelsvideo retrievalmultimodal retrievalsemantic rankingtemporal alignmentrelevance estimationdiversity optimizationframe selectionlong videosstreaming videos
Authors
Wei Ao, Lan Wang, Vishnu Naresh Boddeti
Abstract
The performance of vision-language models (VLMs) in video understanding declines with increasing video duration, as video moments unrelated to the query confuse their language components. Multimodal retrieval has emerged as a critical component of video understanding, addressing this challenge by localizing key visual evidence. However, existing multimodal retrieval methods suffer from biased relevance estimation, limited diversity, and temporal collapse. In this paper, we propose QSVideo, a unified framework that systematically addresses relevance, diversity, and temporal modeling in video retrieval. We first introduce a query-conditioned semantic ranker, QSRanker, which reformulates arbitrary questions into retrieval-friendly queries and estimates structured relevance along object, action, and location dimensions. Building upon this, we design QSRetrieval to jointly optimize relevance and diversity for more informative frame selection. Moreover, we propose temporal alignment strategies tailored for both long and streaming videos to improve evidence recall. Extensive experiments on long and streaming video benchmarks demonstrate that QSVideo greatly enhances video VLM performance under strict frame limit constraints. The code is available at https://github.com/human-analysis/QSVideo.