VisLens: Single-Pass Interpretable Visual Search for Multimodal LLMs
2026-08-31 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors identify that current large multimodal language models have trouble quickly finding small or rare objects in big pictures. They introduce VisLens, a new method that reads early internal signals in the model to spot relevant areas in one go, avoiding slow repeated checks. VisLens then zooms into those parts and combines this with the full image to give a final answer faster. Their approach is much quicker than previous methods while keeping accuracy similar or better.
Multimodal large language modelsVisual searchLogit lensHidden statesReinforcement learningAttention mechanismsInference latencySemantic decodingHigh-resolution imagesModel interpretability
Authors
Jingyi He, Sanghwan Kim, Zeynep Akata
Abstract
Multimodal large language models (MLLMs) struggle with fine-grained Visual Search, the task of locating small or rare objects in high-resolution images. Existing remedies fall into two families: (1) Training-free methods based on attention or confidence scores are accurate but slow, since they require multiple MLLM queries per example. (2) Reinforcement Learning (RL) trained tool-use models are faster at inference but opaque, since their tool calls remain uncontrollable and hard to interpret. To overcome this, we propose \emph{VisLens} (Visual Focus via Logit Lens), a Visual Search method built on the logit lens, which decodes the semantics held in a hidden state by projecting it through the LLM head. VisLens further uses a lightweight tuned-lens that maps early hidden states into the final hidden state space, so visual tokens can be read out from early layers. These tokens are matched to target words in the query to generate a crop of the relevant region, which is fed back in alongside the original image to produce the final answer. The whole process, from decoding to the final answer, completes in a single forward pass without repeated queries. VisLens matches or exceeds prior baselines while delivering a substantial latency advantage, running $8.5$--$9.9\times$ faster than Thyme and up to $22.2\times$ faster than training-free multi-pass search methods.