Test-Time Training for Modality Order Consistency in Vision-Language Models
2026-07-22 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionComputation and Language
AI summaryⓘ
The authors discovered that vision-language models perform differently depending on whether the image or the question is shown first, even though the order shouldn't matter. They designed a training method that fixes this inconsistency during testing, making the models more reliable. Surprisingly, this method also improved the better-performing order, helping both ways get better together. They pinpointed the problem to a specific part inside the model where the internal understanding changes based on the order, and their method helps align this part. Overall, they showed that a simple adjustment during testing can reduce this order sensitivity and boost model performance.
vision-language modelsmodality ordertest-time trainingactivation patchingmodel representationspromptingmid-network layerasymmetric adaptation
Authors
Aditi Gupta, Yossi Gandelsman
Abstract
We find that vision-language models are sensitive to a specific semantically irrelevant change: the order in which the image and question are presented. Across three models and three benchmarks, image first prompting consistently outperforms question-first prompting, revealing a repeatable modality order failure. We use this gap to design an order-consistent test-time training method. Our method substantially closes the modality-order gap across all evaluated settings. Surprisingly, it also yields consistent improvements in the stronger image-first branch over the baseline, hence bootstrapping both orderings toward mutual consistency. Activation patching localizes the ordering failure to a narrow mid-network region where representations diverge sharply between prompt orders. We find that the test-time training method repairs this misalignment across layers. Together, our results identify modality-order sensitivity as a circuit-level failure in VLMs and demonstrate that simple, asymmetric test-time adaptation can effectively mitigate it and even improve performance over the baseline.