Video understanding improves by mixing low and high resolution frames
Rethinking Long-Video Efficiency: A Joint Allocation Perspective on Frames, Pixels, and Front-End Latency
Computer Vision and Pattern Recognition
Summary
Understanding very long videos efficiently is hard because computers must choose which parts of the video to pay attention to. The authors found that using many low-resolution frames combined with a few high-resolution ones works better than only picking a few high-resolution frames. They built a simple method called LoHi that mixes these two types of frames and shows it greatly improves accuracy and speeds up processing for long videos. This approach helps computer models understand videos faster without needing extra training.
What this means in practice
- •For video platform engineers: Process hour-long videos faster and more accurately by combining dense low-res and sparse high-res frames for content understanding.
- •For surveillance system developers: Reduce latency when analyzing long surveillance footage by using mixed-resolution frame selection guided by video metadata and visual features.
Authors
Sixun Dong, Wei Li, Andong Deng, Qi Qian, Victor Zhu, Zhengping Ji, Chen Chen
Abstract
Efficient long-video understanding with vision-language models (VLMs) is often framed as selecting informative frames or visual tokens at a fixed native resolution. We show that per-frame resolution can instead be traded for denser temporal coverage, while front-end decoding latency depends on the size of the candidate pool rather than the final token budget. An empirical study across multiple VLMs and long-video benchmarks yields three findings: dense low-resolution sampling outperforms sparse native-resolution sampling at matched token budgets; resolution-sensitive tasks benefit from selected high-resolution frames; and front-end decoding dominates wall time for hour-long videos. Motivated by these findings, we introduce LoHi, a training-free, single-pass framework that combines a dense low-resolution video stream with sparse high-resolution image frames through the VLM's native video and image pathways. LoHi-Anchor selects high-resolution frames using codec I-frame metadata, while LoHi-SemDiv uses query relevance and visual diversity over CLIP features. Across three long-video benchmarks, LoHi improves average accuracy by 10.6 percentage points over the native-resolution baseline at a matched token budget and by 5.2 percentage points over the strongest prior efficiency method. It also reduces front-end decoding latency by up to 7x on hour-long videos. Project page: https://sixundong.com/projects/lohi