LocalAgent improves object matching by combining global and local checks
LocalAgent: Collaborative Agentic Verification for Fine-Grained Instance-Level Consistency
Computer Vision and Pattern Recognition
Summary
People want computers to tell if two pictures show the exact same object, even if the pictures look different because of lighting or angle. This is hard because the computer must look at the whole scene and also tiny details. The authors created LocalAgent, which uses a big AI model to find suspicious spots and special small tools to carefully compare those spots. Their approach checks fewer places but makes better decisions about whether the objects match.
What this means in practice
- •For robotic vision teams: Improve robot object recognition by verifying exact matches under varying viewpoints and lighting conditions for precise manipulation tasks.
- •For video surveillance operators: Enhance identity confirmation by accurately verifying objects or items across different camera views or times.
Authors
Jiachen Guo, Xinshan Zhu, Boyun Wang, Yanyan Liang, Jun Wan, Feng Lin, Ajian Liu
Abstract
Multimodal large language models (MLLMs) have made rapid progress in general visual understanding, yet they still struggle with fine-grained instance-level consistency verification, where the goal is to determine whether two images depict the exact same physical object under varying viewpoints, illumination, and backgrounds. This task requires both global semantic perception and precise examination of subtle local evidence, since semantically similar objects may differ only in small suspicious regions. To address this challenge, we propose LocalAgent, a collaborative agentic framework that integrates the complementary strengths of a generalist MLLM and lightweight specialist verifiers. The MLLM first reasons globally to identify suspicious local regions, which are then delegated to a Local Evidence Verifier (LEV) toolkit for fine-grained appearance and geometric matching. LEV comprises Local Appearance Consistency (LAC), which estimates appearance similarity under illumination and color changes, and Local Geometric Correspondence (LGC), which estimates spatial correspondence under viewpoint changes. LEV returns structured scalar evidence that the MLLM integrates for the final decision. We further introduce an auxiliary-capped reinforcement fine-tuning (RFT) strategy, in which answer correctness dominates auxiliary tool-use and format signals, enabling the policy to learn when verification is necessary. Experiments show that LocalAgent improves mean accuracy by 4.9% over the strongest CoT-RFT baseline while invoking a verifier on only 12.13% of samples and preserving the general multimodal capabilities of the underlying MLLM. The open-source repository is available at https://github.com/Firefly-zoo/LocalAgent-main.