SeGDeP improves image segmentation by separating meaning and location cues
SeGDeP: Semantic- and Geometric-Aware Decoupled Prompts for Reasoning Segmentation
Computer Vision and Pattern Recognition
Summary
Segmenting images based on language instructions is difficult because it involves understanding both what is described and where it is in the image. The researchers developed SeGDeP, a system that processes meaning and location separately before combining them to create precise image masks. This approach helps identify whether errors come from misunderstanding the description or misplacing the target in the image. Their method showed strong results on benchmark datasets while only modifying a small part of the language model. Tests confirmed that the system learns to handle semantic and geometric information independently and effectively.
language modelimage segmentationsemantic featuresgeometric projectionmask decoderDETRSAMreward-decoupled policy optimizationRefCOCO datasetLoRA
Authors
Linnan Zhao, Xu Liu, Lingling Li, Licheng Jiao, Fang Liu, Wenping Ma
Abstract
Reasoning segmentation converts an implicit linguistic conclusion into a precise mask, requiring both semantic identification and spatial grounding. Existing MLLM-segmenter interfaces either use a special trigger or compress both signals into one context, although they receive different supervision and fail differently. This coupling obscures whether a failure arises from target interpretation or from localization. We present SeGDeP, an explicit what-where interface. A semantic prompt branch and an independent geometric projection path transform resolved MLLM states into semantic features and a DETR-predicted box, which jointly condition a SAM 3 mask decoder. Training first aligns this executable interface, then uses group reward-decoupled policy optimization (GDPO) to balance format, box-IoU, and mask-IoU feedback. SeGDeP-4B reaches 82.7 average cIoU over eight RefCOCO-family splits and 66.0/59.6 gIoU on ReasonSeg val/test while adapting only 0.38% of Qwen3-VL parameters through LoRA. Controlled stage-wise ablations, gradient diagnostics, and prompt interventions further show that the two paths develop complementary semantic and geometric specialization rather than duplicating the same evidence.