AI summaryⓘ
The authors studied how vision-language models (VLMs) decide if they are confident enough to answer a question or should defer, especially when using limited memory. They compared small and large models, both in full precision and compressed (4-bit quantized), and looked at two ways models show confidence: internally (how sure the model is) and verbally (how confident it sounds in natural language). Larger models had much better internal confidence but didn’t express this well verbally, and quantization hurt confidence signals but hardly affected accuracy. Their recommendation is to use larger models with quantization for the best balance of accuracy and detecting mistakes under a fixed memory budget.
Vision-language modelsConfidence signalQuantizationModel scaleError-detection AUROCSelective predictionMean token probabilityCalibration errorMemory budgetNatural language processing
Abstract
Vision-language models (VLMs) deployed on consumer hardware must decide when to answer and when to defer, and that decision depends on having a confidence signal that tracks correctness. A practitioner with a fixed memory budget faces a choice between a small model at full precision, the same small model quantized, and a larger model quantized into the same footprint -- three configurations that push the confidence signal in opposing directions. We measure, on identical inputs, how model scale and 4-bit quantization affect two confidence signals in the Qwen2-VL family: the confidence a model states in natural language, and its own mean token probability over the answer it generates. Across 5,700 predictions spanning six realistic photographic degradations at three severities, we find that scale sharply improves the model's internal uncertainty signal (mean error-detection AUROC 0.80 to 0.98 from 2B to 7B) while its verbalized confidence stays weak and often at chance (mean 0.61 to 0.69): the gap between what the model knows and what it says widens rather than closes with size. We find that 4-bit quantization is nearly free for accuracy (-1.6 points) but expensive for the confidence signal (internal AUROC 0.95 to 0.80, and the verbalized-confidence parse rate collapses from 99% to 64%). For a fixed memory budget the recommendation is therefore to prefer a larger quantized model over a smaller full-precision one: 7B-4bit gives both the best accuracy and the best uncertainty signal (internal AUROC 0.98) of the three configurations that fit. We frame the results as selective-prediction operating points so they translate directly into a deployment recommendation, and we argue that error-detection AUROC, not calibration error, is the metric that exposes the difference between the two signals.