Multi task visual grounding improved by separating then combining features
DeCo: Efficient Decouple-to-Couple Learning for Multi-Task Visual Grounding
Computer Vision and Pattern Recognition
Summary
This paper tackles the challenge of teaching computers to understand language and images together for tasks like locating and outlining objects in pictures. The authors introduce a new method called DeCo that first separates the computer's understanding of different tasks to avoid confusion and then cleverly combines helpful clues from simpler tasks to improve harder ones. Their approach uses less training effort by building on a fixed language-image model, yet performs very well on diverse visual grounding tasks, including natural images and remote sensing data. This balance of independence and cooperation in learning leads to better overall results.
What this means in practice
- •For computer vision engineers: Design multi-task models that better locate and segment objects in images using semantic and spatial cues without heavy retraining of large models.
- •For remote sensing analysts: Improve object detection and delineation in satellite images by integrating task-specific and spatial attention mechanisms as shown by this method.
Authors
Xiaoqiang Lu, Licheng Jiao, Long Sun, Yuting Yang, Xu Liu, Lingling Li, Wenping Ma, Fang Liu
Abstract
Multi-task visual grounding requires models to jointly understand linguistic semantics and perform accurate visual localization and segmentation. Despite the success of multimodal large language models, effectively adapting them to multiple grounding objectives remains challenging. Existing methods commonly enforce task cooperation through shared representations, while overlooking the intrinsic conflict between task-oriented feature interests. In this paper, we introduce $\textbf{DeCo}$, an efficient $\textbf{De}$couple-to-$\textbf{Co}$uple learning framework that resolves this dilemma through a two-stage paradigm: task-specific representation decoupling followed by complementary prior coupling. Specifically, we first propose Task-aware Semantic Decoupling (TSD) to route shared visual cues into individual features under salient word-level guidance, alleviating representation interference between localization and segmentation. Furthermore, we observe that segmentation naturally provides informative localization priors due to dense supervision. Based on this insight, we introduce Hybrid Prior Coupling (HPC), which integrates sentence-level semantic prior with mask-derived spatial prior for enhanced grounding. Built upon a frozen multimodal encoder, DeCo requires lightweight trainable parameters while achieving strong generalization across multiple grounding objectives. Extensive experiments on RefCOCO/+, G-Ref, ReferIt, Flickr, DIOR-RSVG, SARVG1.0, RRSIS-D, RIS-LAD, and RefDIOR demonstrate that DeCo achieves state-of-the-art performance on both natural and remote sensing benchmarks. The code and models are available at https://github.com/xiaoqiang-lu/DeCo.