Beyond Single Expert: Harmonizing Diverse Visual Priors in MLLMs for Spatial Understanding
Computer Vision and Pattern Recognition
Summary
The authors explore how combining different pre-trained models can improve multimodal large language models' (MLLMs) ability to understand spatial information. They found that different models provide unique helpful clues, so they created ViPS, a system that efficiently gathers and mixes these clues to help MLLMs better understand space. Their approach uses a smart proxy to get multiple visual hints with low extra work and a method to combine them in a way that adapts to the situation. Tests show that ViPS improves performance on tasks involving complex spatial reasoning and 3D understanding.
Authors
Xiao Lin, Xiaohu Huang, Kai Han
Abstract
Multimodal Large Language Models (MLLMs) have demonstrated substantial promise in spatial understanding. Existing works typically incorporate prior knowledge extracted from a pre-trained foundation model to further enhance the spatial awareness of MLLMs. In this paper, we first reveal that when integrating diverse foundation models into MLLMs, different models provide complementary spatial priors that benefit different tasks. Motivated by this, we propose $\textbf{ViPS}$, a novel multi-model prior framework designed to fully unleash the potential of incorporating multiple $\textbf{Vi}$sual $\textbf{P}$riors from diverse models into MLLMs for $\textbf{S}$patial understanding. Specifically, ViPS introduces an Efficient Prior Proxy to generate multiple foundational priors with minimal inference overhead, and a Dynamic Prior Fusion mechanism to achieve harmonious and context-aware prior fusion and injection from the prior proxies. Extensive experiments demonstrate that ViPS successfully harmonizes diverse visual priors, establishing new state-of-the-art performance across multiple complex spatial reasoning and 3D spatial understanding benchmarks. Project page: https://visual-ai.github.io/vips