FlashGaze cuts video processing time by skipping redundant patches
FlashGaze: Training-Free Multi-Scale Patch Pruning For Efficient Video Understanding
Computer Vision and Pattern Recognition
Summary
Processing long, high-resolution videos is tough because many parts are similar and don’t add new information. The authors present FlashGaze, a way to quickly spot and skip these repetitive parts before heavy video analysis starts, without needing extra training. This method uses differences in pixel images to decide what to keep or drop, speeding up video understanding while keeping accuracy high. FlashGaze lets video models handle longer videos or higher resolutions on the same hardware.
What this means in practice
- •For video analytics engineers: Improve real-time video analysis by reducing unnecessary data processing while preserving recognition accuracy.
- •For hardware optimization teams: Optimize GPU memory and computation to support processing of longer and higher-resolution videos efficiently.
Authors
Ziye Zhu, Yanghao Zhou, Lixing Tan, Jialiang Kang, Shuxuan Li, Xiao Yang
Abstract
Multimodal Large Language Models (MLLMs) have demonstrated strong performance in video understanding, yet efficiently processing long, high-resolution videos remains challenging. Such videos often contain substantial spatiotemporal redundancy, and processing redundant visual tokens can incur avoidable computational overhead. Many existing methods prune visual tokens during or after vision transformer (ViT) encoding, leaving much of the encoding cost unaddressed. Some approaches prune patches before encoding but rely on learned auxiliary networks for patch selection, incurring additional training and inference overhead. To address these limitations, we propose FlashGaze, a training-free method that reduces spatiotemporal redundancy before ViT encoding without introducing auxiliary networks. FlashGaze uses pixel-space differences as a proxy for information loss and employs Quadtree Dynamic Programming to jointly optimize patch dropping, merging, and keeping under a fixed budget. Experiments on two MLLM backbones across multiple benchmarks demonstrate substantial efficiency gains while largely preserving accuracy. On Qwen3-VL-8B, FlashGaze retains 98% of the full-input baseline accuracy on LongVideoBench while achieving up to 5.4x and 17x speedups in ViT encoding and MLLM prefill, respectively, and reducing peak GPU memory usage by a factor of 1.8. These efficiency gains enable the model to process videos with more frames and higher resolutions on the same GPU hardware, unlocking video understanding at scales previously out of reach.