Latent lens improves visual understanding in small language models
Latent-Lens: Visual Perception in Small Language Models Through Latent Communication
Machine LearningArtificial Intelligence
Summary
Small language models struggle to understand images well because they usually get visual information as plain text, which loses important details. The authors introduce Latent-Lens, which lets a visual model and a small language model share information in a hidden, compact form rather than text. This method helps the language model better answer questions about images and describe them more accurately. The approach works with different model types and only needs a small amount of extra training.
What this means in practice
- •For ai application developers: Enhance small language models with richer visual input for better performance in image-based question answering and description tasks.
- •For mobile device engineers: Build efficient vision-language features on resource-limited devices by enabling small models to communicate visual data without large text bottlenecks.
Authors
Mo Shahdloo, Faegheh Sardari, Mathew Salvaris
Abstract
Bridging independent vision-language models (VLMs) and small language models (SLMs) in a modular multi-agent system typically forces them to exchange information as discrete text, which creates a lossy bottleneck for spatial and fine-grained details. This constraint is especially acute for SLMs, whose limited capacity makes them more reliant on the quality of the input signal. Recent latent-communication methods exchange states between language models. How a VLM can pass visual information to a separately pretrained text-only SLM though remains unexplored. We propose \textbf{Latent-Lens}, a lightweight codec that bridges the representation gap between a frozen VLM and a frozen SLM directly in latent space, enabling them to communicate purely through hidden states. The codec consists of a domain encoder that projects hidden states from the VLM into the SLM's embedding space, and an attention decoder formalised as a LoRA that teaches the SLM to attend to these visual prefixes, while training only ${\sim}1.6$M parameters (0.2\% of the combined model size). On visual question answering (CLEVR, GQA), Latent-Lens achieves $81.0\pm7.9$\% and $65.4\pm5.4$\%, compared with $58.3\pm5.7$\% and $61.8\pm1.0$\% for text communication with the same LoRA budget (+30.2--69.8pp relative to frozen text communication), while on image captioning (Flickr8k) it improves over the same baseline by +16.5 (+18.8 frozen). We also evaluate Latent-Lens with TinyLlama-1.1B and InternVL2-1B, which use different architectures, vocabularies, and hidden dimensions.