Latent diffusion approach improves multimodal reasoning and robot tasks
Uni-LaDiR: Latent Diffusion Unifies Multimodal Reasoning
Machine LearningComputation and Language
Summary
Multimodal reasoning means thinking that combines different types of information, like pictures and text. Many current systems just stick these different kinds of information together and try to reason over them, which can be confusing for the model. The authors created Uni-LaDiR, a new way that puts all ideas into one shared space, making it easier for the model to understand and predict the next steps of reasoning. This approach helps the model get better at visual reasoning and at controlling robots to perform tasks. It was tested on many benchmarks and showed improved accuracy compared to previous methods.
What this means in practice
- •For computer vision engineers: Enhance image and language understanding systems by unifying reasoning steps into shared representations for improved visual reasoning accuracy.
- •For robotics developers: Improve robotic manipulation by predicting reasoning steps in a shared latent space, leading to better task performance from visual and language inputs.
Authors
Haoqiang Kang, Yizhe Zhang, Nikki Lijing Kuang, Yian Ma, Lianhui Qin
Abstract
Multimodal reasoning requires models to draw on information from multiple modalities throughout the reasoning process. Yet existing methods often concatenate modality-specific thought tokens in a single sequence, leaving the model to bridge representational differences as it reasons across modalities. We introduce Uni-LaDiR (Unified Latent Diffusion Reasoner), a framework that brings these thoughts into a shared latent space for reasoning. A unified encoder maps teacher reasoning steps from different modalities into shared thought tokens, trained to preserve the information needed for later reasoning steps and the final answer or action. Because the same context can support multiple valid next steps, we use diffusion to predict the next block of thought tokens from the input and preceding blocks. Jointly training the encoder and diffusion reasoner with shared model weights encourages thought tokens to be both useful for the task and predictable from the available context. At inference, the model generates these tokens without teacher observations. Across eleven vision-language model (VLM) benchmarks and two vision-language-action (VLA) suites, Uni-LaDiR achieves relative gains over the strongest evaluated baselines of 7.3% on visual reasoning tasks and 6.1% on robot manipulation tasks.