Vision language models cut costs by smartly cropping or compressing images
C$^2$-Routing: Complementary Cropping and Compression for Efficient Vision-Language Inference
Computer Vision and Pattern RecognitionMachine Learning
Summary
Answering questions about images with AI usually takes a lot of computing power. The authors found that using either cropping or compressing images alone isn’t always enough because each method keeps different important parts of the picture. They created a system called C²-Routing that decides when to crop or compress images to save computing resources while keeping answers accurate. This approach also includes a safety step to avoid losing important information. As a result, this saves a lot of computing effort with almost no loss in answer quality on many tests.
What this means in practice
- •For mobile app developers: Build mobile apps that answer questions about photos efficiently by reducing image data with adaptive cropping or compression to save device resources.
- •For cloud service operators: Cut computing costs and energy usage when running image-based AI query services by smartly selecting data reduction methods per image.
Authors
Guorui Lu, Qinyu Chen
Abstract
Visual Question Answering (VQA) imposes substantial computational overhead on vision-language models (VLMs), motivating model-agnostic input reduction methods that can be readily applied across diverse VLMs, including in black-box settings. Simply enforcing a visual-token cap reduces computation, but it can discard information needed to answer the question. A single reduction strategy also struggles to accommodate different scenarios. Our key observation is that cropping and compression are complementary: cropping highlights small targets and local details, whereas whole-image compression better preserves context for large targets or distributed evidence. Based on this observation, we propose C$^2$-Routing, which adaptively selects cropping or compression according to the input image and question. Since both strategies may still discard key information, we further design a safety gate that uses the predicted joint correctness distribution under original and reduced inputs to guide fallback and protect accuracy. To accommodate consumer GPU resource constraints, all training and evaluation experiments run on RTX 3090 GPUs. To measure gains beyond token capping, we use full-image baselines with the same cap of 1,024 visual tokens. We conduct extensive evaluation on three VLMs and 18 datasets spanning local details, global scenes, scene text, and documents. With the visual budget already capped, C$^2$-Routing further reduces end-to-end FLOPs by 36.53% and visual tokens by 42.36% on average, with a mean relative task-score decrease of only 0.77%.