Visual lookup memory boosts vision language model capacity efficiently
V-LUMEN: Visual Lookup Memory for Embedding Scaling in Vision-Language Models
Machine LearningArtificial IntelligenceComputation and LanguageComputer Vision and Pattern Recognition
Summary
Vision-language models struggle to scale their understanding of images and text simultaneously because images are complex and continuous, unlike text which is discrete. The authors designed V-LUMEN, a method that stores visual information in a special memory bank that can be quickly looked up without increasing the model size or processing time. By converting image features into fixed codes and organizing them spatially, V-LUMEN helps the model remember and use more visual details for reasoning tasks. This approach improves performance on several tests and can be transferred to new image sets without retraining.
What this means in practice
- •For machine learning engineers: Increase vision-language model effectiveness on reasoning tasks without larger models or more computation by integrating visual lookup memory.
- •For ai system builders: Implement visual memory modules that enable scalable image understanding while maintaining fast inference and adaptable model size.
Authors
Miso Choi, Daekeun Kim, Eunji Kim, Jungbeom Lee
Abstract
Embedding-level scaling expands the capacity of language models through lookup-based memory rather than by increasing model depth, width, or dense computation. Extending this idea to vision-language models (VLMs) is challenging because visual features are continuous, spatially structured, and query-dependent, unlike discrete text tokens that naturally support n-gram lookup. We introduce V-LUMEN, a visual lookup memory for VLMs. V-LUMEN discretizes visual features with a frozen codebook, builds spatially aware memory keys through spatial aggregation-based hashing, and modulates them with text-conditioned relevance signals; all addressing is non-parametric, and the retrieved embeddings are fused into the backbone through a gated residual pathway. On five reasoning-intensive benchmarks, V-LUMEN substantially improves a frozen Qwen3-VL-2B backbone and transfers zero-shot to three benchmarks whose images are disjoint from the training data. At a matched parameter budget, moving part of a LoRA or MoE adapter's capacity into V-LUMEN preserves accuracy on two backbones, indicating that external visual memory is an independent capacity axis rather than a stronger adaptation method. Because lookup cost is constant in table size and incurred only once at prefill, this capacity grows in storage rather than in per-token compute. Code is available at https://github.com/miso-choi/V-Lumen.