Language reasoning vectors restore vision language model thinking power
Can Linguistic Reasoning Vectors Enhance Multimodal Reasoning Ability?
Artificial Intelligence
Summary
Vision-language models combine image and text understanding but often lose some of the language reasoning skills from the original language-only models. The authors show that pulling reasoning signals from the original language model and adding them back into the multimodal model helps it think more clearly without retraining. They create small vectors called Reasoning Vectors that capture how reasoning changes answers and insert these into the vision-language model’s language parts. This helps improve reasoning on tasks involving both images and language while keeping the main model unchanged.
What this means in practice
- •For multimodal ai developers: Improve reasoning ability in vision-language models without retraining their core components by integrating reasoning vectors from language-only models.
- •For augmented reality software teams: Enhance multimodal understanding in AR devices by restoring language reasoning to support clearer interaction between text and images.
Authors
Ziyi Wang, Li Li, Aolin Zhou, Yankun Shen, Chonghan Liu, Shuxia Lin, Xu Yang
Abstract
Most Vision-Language Models (VLMs) are built by extending pretrained Large Language Models (LLMs) with visual modules and multimodal alignment. However, this multimodal scaling often degrades the language-side reasoning ability originally encoded in the base LLM. While the base LLM retains usable reasoning after scaling, the aligned VLM itself cannot reliably access this ability. Therefore, recovering the degraded reasoning capability in VLMs would benefit more from seeking help from the base LLM than from the VLM alone. Motivated by this, we propose LIFT (Language-side reasonIng Facilitation and Transfer), a lightweight vector-intervention method that transfers reasoning capability from the base LLM to the VLM without retraining the backbone. LIFT defines Reasoning Vectors as answer-token hidden-state differences between a Reasoner path with an explicit reasoning trace and a Solver path without it, and injects these vectors into language-side activations of the target VLM. LIFT further supports learnable vector adaptation while keeping the VLM backbone frozen. We evaluate LIFT on two VLMs across six reasoning benchmarks, comparing Reasoning Vectors extracted from the base LLM and from the aligned VLM under matched protocols. Results show that LLM-derived vectors consistently outperform VLM-derived vectors, confirming that the base LLM is a more effective source for recovering reasoning. LIFT partially recovers degraded reasoning through lightweight language-side interventions. Further analyses show that Reasoning Vectors influence intermediate reasoning behavior rather than merely altering final answers. The source code will be released soon.