Visual grounding improved by stepwise region refinement and editing
CoEvolve: Construct-to-Edit Visual Grounding with Bidirectional State Refinement
Artificial IntelligenceComputer Vision and Pattern Recognition
Summary
Visual grounding means finding and drawing a box around the object described by words in an image. Most current methods try to find the object in one step, which can make it hard to fix mistakes if the box is wrong. The authors created CoEvolve, a method that first picks candidate regions step-by-step and then refines these regions by editing their boundaries. This approach helps catch and correct errors better, making the final boxes more accurate even with tough images.
What this means in practice
- •For computer vision engineers: Build image search and object detection systems that improve accuracy by incrementally refining object locations described by language.
- •For geospatial analysts: Use refined visual grounding to better locate objects or areas of interest in remote-sensing images from textual descriptions.
Authors
Dongwei Sun, Yujie Zhang, Bowen Yao, Pei Liu, Jing Yao, Xiangyong Cao
Abstract
Visual grounding localizes an object described by language with a bounding box. Most multimodal grounding models compress target identification, spatial reasoning, and boundary estimation into one terminal prediction. Free-form rationales make reasoning linguistically explicit but do not necessarily expose measurable, editable spatial states. Intermediate localization errors are therefore difficult to diagnose and correct, allowing incorrect region choices and imprecise boundaries to persist in the final box. We introduce CoEvolve, a construct-to-edit framework that separates grounding into explicit state construction and state editing. Region-Evolution Reinforcement (RER) organizes grounding analysis into a progressive semantic--spatial trajectory, with each reasoning step committing to an explicit candidate region. Bidirectional Denoising Refiner (BDR) treats the reasoning text as fixed semantic context and refines the trajectory's coordinate fields through bidirectional same-position reconstruction. Geometry- and behavior-level objectives provide target geometry and edit-preference signals for consolidating reliable candidates, preserving accurate inputs, or correcting toward annotations. Evaluations cover natural-image and remote-sensing grounding. With a 9B backbone, CoEvolve rivals models up to 241B parameters in grounding accuracy. Under controlled corruption, a single BDR pass improves mean box overlap by over 27 percentage points, demonstrating strong recovery from substantial localization errors. State-source comparisons further support the complementarity of explicit state construction and source-matched editing. The project is at https://sundongwei.github.io/CoEvolve_Project/.