Multimodal models improve by connecting vision and language layers dynamically
Where to Connect? Boosting MLLMs via Dynamic Gated Pathways across ALL ViT and LLM Layers
Computer Vision and Pattern Recognition
Summary
Multimodal Large Language Models (MLLMs) combine vision and language but often use a fixed connection point, missing out on how different language understanding layers need different kinds of visual information. The authors create a method called Dynamic Gated Pathways (DGP) that lets every layer of a vision transformer (ViT) connect to every layer of a language model, with each language layer choosing which visual features to focus on depending on its needs. This flexibility improves the model’s performance on various vision-language tasks and provides insight into which types of visual information each language layer uses.
What this means in practice
- •For multimodal ai developers: Enhance multimodal AI systems by implementing dynamic connections between vision and language model layers to improve understanding in complex tasks combining images and text.
- •For ai system engineers: Integrate adaptive visual information routing in language models to increase accuracy and flexibility of vision-language applications like visual question answering and image captioning.
Authors
Yingying Yan, Jiaqi Tang, Wei Wei, Qianzhou Wang, Lei Zhang, Jianmin Chen, Yuyang Xia, Botong Geng, Jinjian Wu, Qifeng Chen
Abstract
A central yet under-explored question in Multimodal Large Language Models (MLLMs) is where to connect---how visual information from a hierarchical vision encoder should be wired into the layer-wise semantics of a large language model. Mainstream MLLMs inject a fixed single-layer visual representation into the LLM and treat all LLM layers as a single consumer, ignoring that different layers demand different visual granularities. Recent multi-layer fusion methods either compress hierarchical features on the encoder side or rely on predefined sparse or hierarchical layer-to-layer connections, and therefore stop short of modeling adaptive cross-layer interactions between the ViT and LLM hierarchies. We address this gap with DGP (Dynamic Gated Pathways), a routing mechanism that establishes all-to-all connections between every ViT layer and every LLM layer, with each LLM layer dynamically gating its preference over visual layers conditioned on its own semantic state. Built on top of LLaVA-1.5-7B and 13B, DGP delivers consistent gains over the LLaVA-1.5-7B baseline (e.g., +4.2 on SQA$^{I}$, +3.3 on MM-Vet, +3.1 on VizWiz and LLaVA$^{W}$) and surpasses prior multi-layer visual fusion methods on most benchmarks. Layer-wise routing analyses further show that different LLM layers do select different visual granularities on demand, supporting both the effectiveness and interpretability of the proposed dynamic gated pathways. The code is available: https://github.com/YingyingYan1/DGP.