Compression changes vision language model scores without changing answers
BEFORE THE FLIP: Measuring Hidden Score Shifts In Quantized Vision Language Models Before The Answer Changes for Visual Question Answering
Computer Vision and Pattern Recognition
Summary
Compressing vision language models to use fewer bits can keep their answers the same but still change the confidence scores behind those answers. The authors created a way to measure these hidden score shifts by comparing compression effects to changes caused by replacing image parts. They found that lower-precision compression shifts scores more but rarely changes the actual answers given. Trying to adjust bit precision for each question did not consistently improve results. This suggests that compression can quietly alter how models weigh options without flipping their final choices.
What this means in practice
- •For machine learning engineers: Optimize storage and computation for vision language models aware of hidden score shifts without losing answer correctness.
- •For ai system testers: Develop tests to detect subtle confidence changes in compressed models that do not affect final answers but impact reliability.
Authors
Sourajit Saha, Shubhashis Roy Dipta, Shaswati Saha, Nobin Sarwar, Yuxuan Jiang
Abstract
Quantization makes vision language models (VLMs) cheaper to store and run by using fewer bits to represent their weights. While unchanged answers on visual question answering (VQA) after compression are an expected behavior, they can still hide changes in the underlying scores (log probabilities). For example, a model may still answer yes after compression, even as the score gap between yes and no shrinks. We introduce BEFORE THE FLIP to measure these hidden changes. Our method compares the score change caused by compression with the change caused by replacing the image's internal representations, or image tokens, with one fixed average token. We then increase the precision of one weight group at a time to identify where extra bits help, and test whether choosing different groups for each question offers benefits beyond shuffled controls. Among 8,277 LLaVA questions where image token replacement measurably affects the scores, 4-bit compression shifts the yes or no score gap farther toward the replacement output than 8-bit compression. Qwen shows the same pattern, but with a smaller difference. Yet only 265 of 9,000 LLaVA answers change at 4 bits. In a separate study of 1,024 calibration questions, choosing weight groups separately for each question does not outperform both shuffled controls at any tested storage budget. These findings show that compression can alter the scores behind unchanged answers, but do not establish a reliable benefit from adjusting precision for each question.