Recompute or Reuse? Diagnosing and Mitigating Textual Shortcuts in VLM Self-Reflection
2026-08-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors studied vision-language models (VLMs) to see if they update their answers when the visual input changes. They found that instead of reanalyzing the new image each time, models often rely on previous text-based reasoning as a shortcut, which can cause them to stick to old answers. This shortcut becomes weaker if parts of the old reasoning are removed or rearranged. To fix this, the authors created a method called Fresh-State Attention Firewall (FSAF) that encourages the models to recompute answers based on the new image, resulting in more accurate updates.
vision-language modelschain of thoughtvisual recomputationtextual shortcutcounterfactual analysisanswer updatingprior reasoningFresh-State Attention Firewallself-reflectionvisual update rate
Authors
Wenxiao Fan, Jingling Fu, Fang Li, Luohang Liu, Yu He, Lichen Ma, Zhiyang Yu, Weishan Bi, Junshi Huang, Yan Li, Gu Simiu, Kan Li
Abstract
Vision-language models (VLMs) are expected to revise their reasoning when visual evidence changes. Failures to do so are often attributed to insufficient visual attention or contextual inertia, leaving unclear what models reuse instead of recomputing from the current image. We show that evidence-bearing reasoning in a prior chain of thought (CoT) can form a textual shortcut that competes behaviorally with visual recomputation. Across 16 VLMs, a matched counterfactual analysis identifies evidence-bearing content as the most robust carrier of prior-CoT influence. Removing this evidence-bearing content shifts answer preference more than removing length-matched non-evidence context or the final-answer span, with prior control weakening progressively as more stale evidence is removed. Reordering this evidence also weakens prior control, showing that its organization modulates shortcut strength. Beyond the immediate answer, the shortcut can retain residual influence after answer correction: weakening current-image support shifts preference back toward the prior answer, while repeated prior answers and reused premises arise mainly when the shortcut remains active. To limit this influence, we introduce Fresh-State Attention Firewall (FSAF), a training-free intervention that isolates fresh computation from the prior CoT. Across five VLMs, FSAF raises visual update rate from 35.28% to 53.61% and reduces prior-answer rate from 39.22% to 3.67%. Reliable VLM self-reflection therefore requires more than looking again: fresh visual recomputation must be protected from stale textual reuse.