LOCI: A Locator-Critic with Refinement Loop
2026-08-31 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors found that Vision-Language Models often make mistakes not because they can't think deeply, but because they can't find the important parts of an image. They created a new method called LOCI that separates the task of finding image clues from checking if those clues actually answer a question. LOCI uses two agents: one to locate possible evidence and another to critique and improve it through repeated steps. This method helped models perform better on challenging image understanding tests without needing extra training.
Vision-Language ModelsVisual groundingEvidence verificationIterative refinementLocator agentCritic agentComplex visual understandingZero-shot learningBenchmark evaluation
Authors
Walid Bousselham, Mathilde Caron, Arsha Nagrani, Cordelia Schmid
Abstract
Vision-Language Models (VLMs) still struggle on tasks requiring complex visual understanding. We argue that the core issue is not high-level reasoning, but instead failing to locate critical details in the image. Due to this shortcoming, VLMs generate often plausible but incorrect reasoning based on flawed perceptual grounding. To address this, we propose Locator-Critic (LOCI), a training-free framework that decouples visual search from evidence verification. LOCI employs a Locator agent to propose candidate visual evidence and a separate Critic agent to evaluate its relevance and sufficiency. These agents engage in an iterative refinement loop, progressively improving the evidence until it is adequate to answer the given question. This decoupled, self-correcting process yields substantial performance gains, achieving state-of-the-art results on multiple complex visual benchmarks. LOCI improves accuracy for both open-weight models like Qwen3-VL (+12.1 on V*, +5.8 on HR-Bench and +11.2 on VisualProbe-Hard) and proprietary models like Gemini 2.5 Pro (+8.9 on V*, +4.3 on HR-Bench, +4.8 on VisualProbe-Hard).