Searching for Task-Specific Vision Paths: Evolutionary Block Pruning Across Vision-Language Models
Computer Vision and Pattern Recognition
Summary
The authors studied if parts of a vision-language model's image processing steps can be safely skipped for different types of questions without retraining the model. They tested ways to choose which blocks to skip, either using one shared skip pattern or customizing it based on question type, like OCR or counting. Their results show that while searching for the best skip patterns improves performance, using capability-specific routes (tied to question type) is less reliable and doesn’t always transfer well across different models or datasets. Overall, they found some improvement by carefully selecting which vision blocks to skip, but capability labels don’t guarantee better or universal routes.
Authors
Tarun Tomar
Abstract
Vision-language models normally execute the same complete vision encoder for every question, even when OCR, counting, object, attribute, and spatial queries may not require identical computation. We study whether fixed-budget combinations of vision blocks can be skipped without fine-tuning. A shared K-block route skips one searched set of exactly K blocks for every question, while a capability-specific K-block policy selects one same-size route using a known capability label. We introduce a source-balanced evolutionary search and compare it with independent ranking, contiguous removal, and random routes at matched budgets. Experiments use Qwen2.5-VL-3B-Instruct, SmolVLM2-2.2B-Instruct, and an 876-example image-disjoint selection split. Search transfers across architectures: on SmolVLM2, the searched shared four-block route beats independent construction by 4.91 percentage points. Capability specialization is less stable. On Qwen, the six-block capability policy beats the shared route by 2.17 points, driven by a 7.10-point OCR gain. On sealed IIIT5K, however, the SmolVLM2 OCR-specific route trails its shared route by 13.6 points. Combinatorial search reliably improves route construction, but capability labels do not define universally transferable vision pathways.