Vision language models reduce unneeded visual calls improving accuracy
When Should a VLM Look? Paying Only for Visual Calls That Were Needed and Used
Artificial Intelligence
Summary
Vision-language models often make guesses about images and sometimes look at parts they don’t actually need, which wastes resources and hurts performance. The authors found that many visual calls were unnecessary and proposed a method called CounterCredit that only pays for image views that help answer questions. This method compares the value of looking at a specific part of an image to guessing without seeing it and rewards the model when visual information is genuinely helpful. This approach led to better accuracy and fewer unnecessary image views on multiple tests.
What this means in practice
- •For ai service developers: Reduce unnecessary image processing in vision-language services to improve efficiency and accuracy of user queries.
- •For robotics system builders: Improve the decision-making of robots that use cameras by enabling them to only analyze visual data when necessary.
Authors
Kunyu Peng, Junming Liu, Ruiqi He, Qingzhuo Wang, Jianzhong Qi, Xianhui Liu
Abstract
Vision-language agents that crop and zoom are trained with rewards that credit a successful tool call, yet a successful call does not show that the model needed to look or used the pixels it received. On our cold-start checkpoint only 10% to 12% of visual calls were both needed and used, and released agents make spurious calls 36% to 87% of the time on individual benchmarks. Outcome rewards, judge rewards, and branch probes each observe one side of this failure, and about two thirds of what an outcome reward pays goes to calls that were neither needed nor used. CounterCredit asks both questions of every image-returning call at its realized pre-call state, using the policy's own gold-answer score. A decision value compares the realized visual branch with answering immediately; an evidence value compares the returned crop with random same-size patches substituted into the same call. A call verified on both earns cashback and every other executed call pays rent; the price is bounded so that every correct trajectory outranks every wrong one, and a dual-channel GRPO advantage keeps the price in its own units. From the same cold start, prompt pool, and budget, CounterCredit reaches 89.5% on V*, 80.2% on HR-Bench-4K, and 76.4% on HR-Bench-8K, 6.3 to 9.4 points above outcome-only GRPO at 1.78 against 1.84 calls per question, and lowers the spurious-call rate to 31% to 36%, the lowest among the agents evaluated. The same recipe lifts a Qwen3-VL-8B base from 75.4 to 80.8 on average.