Vision language models struggle to link confidence with reasoning steps
The Mirage of Calibrated Confidence: Trajectory-Independence of Verbalized Confidence in Vision-Language Models
Artificial Intelligence
Summary
Some vision-language AI models say they are confident in their answers even when their reasoning process is flawed or uncertain. The authors found that these models’ confidence levels don’t reliably depend on the actual steps the models took to reach their answers. They developed new tests to check whether confidence reflects reasoning quality and found many existing methods miss this problem. This work highlights that calibrated confidence can be misleading in current AI evaluation.
What this means in practice
- •For machine learning engineers: Evaluate AI model confidence alignment with reasoning quality to improve model reliability in vision-language tasks.
- •For ai safety teams: Identify misleading confidence outputs by vision-language models to better assess risks in deploying AI systems with self-correction.
Authors
Jisoo Yang, Jaeho Han, Trung X. Pham, Junyeong Kim
Abstract
A calibrated Vision-Language Model (VLM) can repeatedly self-correct, say "Wait, I should recheck," arrive at the wrong answer, and still report high confidence. We find that this occurs because verbalized confidence is largely trajectory-independent in the VLMs and calibration methods we evaluate. We examine this through three complementary lenses: content variation, token masking, and the model's own hesitation markers. We show that confidence is insufficiently sensitive to what the reasoning trajectory actually contains, and that calibration training can paradoxically worsen this disconnect. Since existing metrics like ECE and AUROC cannot detect this problem, we propose the Trajectory-Grounding Score (TGS) in two complementary forms: TGS-self, which compares confidence with and without access to the model's own trajectory, and TGS-pair, which tests whether the model assigns higher confidence to correct trajectories than to flawed ones along the vision, reasoning, and answer axes. We propose TGS-Bench, a model-agnostic suite spanning 10 benchmarks with controlled good/bad trajectory pairs, and show that conventional calibration rankings diverge from trajectory-grounding rankings, exposing a blind spot in current evaluation practice.