DRAgent: Discriminative Reasoning Agent for Referring Expression Segmentation
2026-08-24 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address the problem of identifying and outlining objects in images based on a text description. They point out that current methods sometimes make errors by turning object locations into text tokens. To fix this, they created DRAgent, which first finds candidate object locations using a detector and then uses a language model to choose the right one through a two-step checking process. Finally, they use the chosen location to guide a segmentation model in drawing the exact shape of the object. Their approach shows good results on standard datasets for referring expression segmentation.
Referring Expression SegmentationMultimodal Large Language ModelsObject DetectionVisual-Semantic AlignmentLocalizationSegmentationCandidate SpaceDiscriminative ReasoningLoRA Fine-tuningFoundation Models
Authors
Yujie Qi, Luyan Zhang
Abstract
Referring Expression Segmentation (RES) aims to generate a pixel-level mask for the object specified by a language expression. Recent methods based on multimodal large language models (MLLMs) often rely on one-pass coordinate prediction for visual localization, which serializes continuous spatial locations as discrete text tokens and may lead to localization bias and alignment errors. To address these issues, we propose DRAgent, an MLLM-driven discriminative reasoning (DR) framework for RES. Instead of requiring the MLLM to generate localization coordinates, DRAgent first constructs a detector-generated candidate space and then uses the MLLM as a visual-semantic target discriminator. Specifically, the MLLM performs reliable target selection among potential distractors through a two-stage DR mechanism, which first screens high-recall candidates and then performs instance-wise verification. The selected target box is subsequently used as a spatial prompt for a foundation segmentation model to produce the final pixel-level mask. Furthermore, we construct a self-consistency-filtered reasoning-chain data pipeline for LoRA-based fine-tuning, providing more reliable supervision for enhancing the MLLM's discriminative reasoning capability. Experiments demonstrate that DRAgent achieves competitive performance on RefCOCO, RefCOCO+, and RefCOCOg.