Large vision language models improve speed by smart token trimming
Shallow to Deep: Aligning Token Pruning with Stage-wise Roles in LVLMs
Computer Vision and Pattern Recognition
Summary
Large vision-language AI models process many image parts, which slows them down. The authors found that cutting down this data too early hurts performance because early parts focus on edges while later parts handle meaning. They designed a method called STD that prunes image tokens differently at different stages, keeping important details first and reducing data only when it’s safe. This method speeds up the models a lot and keeps accuracy high.
What this means in practice
- •For mobile app developers: Build faster image understanding features by using stage-aware token pruning to reduce computing costs without losing accuracy.
- •For cloud ai service providers: Deploy large vision-language models more efficiently by integrating stage-adaptive token pruning to reduce latency and resource use in real-time applications.$Commercial implications: Enables cost-effective, faster AI services by reducing required computation, suitable for AI cloud platforms offering vision-language APIs.
Authors
Shuo Zhang, Jintao Tong, Yixiong Zou, Yuhua Li, Ruixuan Li
Abstract
Large Vision-Language Models (LVLMs) incur high computational costs from redundant visual tokens. Although training-free attention-based multi-layer pruning in the vision encoder stage has been explored as an effective strategy, we find that pruning in shallow layers consistently degrades performance. In this paper, we aim to understand this problem and seek a solution. By analyzing attention patterns across network depth, we find that shallow layers primarily function as edge detectors with chaotic attention maps, while deeper layers transition through local subject recognition and unstable semantic aggregation. To address the misalignment between pruning strategies and network stages, we propose STD, a hierarchical token pruning framework that adapts token selection mechanisms to the functional role of each network stage. STD employs High-Frequency Spectral Analysis in shallow layers to deterministically preserve structural edges, uses Gaussian-Smoothed Attention in intermediate layers to maintain spatial coherence, and introduces a Stability-Adaptive Trigger in deep layers to execute pruning only during semantically stable phases. Extensive experiments show that STD outperforms state-of-the-art pruning methods by 1.1% on LLaVA-1.5-7B with 88.9% token reduction, while also being plug-and-play and highly effective when combined with other methods, and by 2.1% on LLaVA-NeXT-7B with 94.4% reduction, delivering a 3.9x speed-up in the prefilling stage. Our code will be released at https://github.com/Twilight03/STD.