Multimodal language models improved by controlling hidden reasoning steps
LatentWeave: Test-Time Latent Optimization for Multimodal Reasoning
Machine Learning
Summary
Multimodal language models combine text and images to answer questions, but they sometimes make mistakes during their step-by-step thinking. The authors introduce LatentWeave, a method that gently guides the model's internal thinking process during answering without changing the model itself. This method helps catch and fix errors early by adjusting hidden states and using special tokens to keep the reasoning on track. Their experiments show this approach improves the model’s performance on various tasks that involve understanding both images and text together.
What this means in practice
- •For multimodal ai engineers: Improve inference accuracy by implementing latent state corrections during model reasoning without retraining the original model.
- •For ai system integrators: Enhance multimodal question-answering systems by stabilizing internal reasoning steps to reduce error accumulation.
Authors
Fanci Meng, Kun Wang, Guibin Zhang, Xinlei Yu, Jiaming Zhang, Ming Li, Yang Wang
Abstract
Recent advances in test-time optimization have shown that the multimodal reasoning performance of large multimodal language models (MLLMs) can be improved during inference without modifying backbone model parameters. However, existing approaches primarily operate at the prompt or output level, leaving the latent reasoning process largely uncontrolled and insufficient to prevent error accumulation during autoregressive generation. To address this limitation, we propose LatentWeave, a test-time framework that treats multimodal reasoning as a controllable latent trajectory. LatentWeave performs structured Latent Weaving, progressively refining intermediate hidden states segment-wise to correct emerging deviations, and further introduces Reasoning Anchors via lightweight WEAVE tokens to stabilize reasoning structure and guide answer-oriented generation. These mechanisms are designed to enable fine-grained, step-aware intervention in the latent reasoning process while keeping the main MLLM backbone parameters fixed. Extensive experiments across diverse multimodal reasoning benchmarks report improvements over the evaluated baselines; the contribution of latent optimization, anchor training, and additional test-time computation is analyzed separately in the ablation and efficiency studies. Our code is available at https://github.com/jins7/LatentWeave.