Foveatree improves video large language model efficiency with smarter token pruning
FoveaTree: Information-Density Trees for Query-Guided Visual Token Pruning in Video Large Language Models
Computer Vision and Pattern Recognition
Summary
Video language models process many small picture pieces per frame, which makes them slow and costly. The authors found that existing ways to cut down these pieces either miss important details or get distracted by irrelevant parts. They created FoveaTree, a two-step approach that smartly decides which picture pieces to keep or merge based on how much detail is lost and how relevant they are to the task. This method keeps key information while reducing workload, tested across several video question-answering and captioning challenges with better results.
What this means in practice
- •For video platform developers: Use FoveaTree to reduce computational costs when running video question-answering or captioning services without losing accuracy.
- •For mobile app developers: Enable more efficient video understanding features on devices with limited resources by pruning visual tokens intelligently during processing.
Authors
Zihan Meng, Weidong Chen, Dexiang Hong, Xinyan Liu, Zhendong Mao
Abstract
Video large language models (Video LLMs) encode each frame into hundreds of visual tokens, making video inference expensive. Existing methods reduce this cost through similarity-based merging or attention-based selection but face two challenges: Hierarchy-blind Pruning, which overlooks regional compressibility and can discard critical evidence, and Attention Drift, where positional effects and attention sinks shift text-to-vision attention away from task relevance. We propose FoveaTree, a two-stage, training-free visual-token pruning framework. First, Merge-Gain Density Estimation measures the feature distortion caused by regional merging and constructs a quadtree-derived density map to guide token dropping, merging, and retention. Second, Attention Tree Pruning uses the surviving units and inherited tree topology to refine text-to-vision attention in intermediate LLM layers, preserving relevant regional evidence while removing low-relevance content. We evaluate FoveaTree on InternVL3.5-8B and Qwen3-VL-8B across five video question-answering benchmarks and one captioning benchmark at 50%, 30%, and 10% visual-token retention. FoveaTree consistently achieves overall state-of-the-art performance across the evaluated settings, offering a strong accuracy-efficiency trade-off.