Language conditioned routing improves locating objects in images
LaMER: Language-Conditioned Multi-Expert Routing for Weakly Supervised Referring Expression Comprehension
Computer Vision and Pattern RecognitionComputation and LanguageMachine Learning
Summary
Figuring out which part of a picture matches a spoken or written description is hard without detailed guidance, especially because different image features like shape or texture matter in different cases. The authors designed a system called LaMER that combines four specialized image readers and chooses how to mix them based on the description, without needing exact example boxes during training. This approach helps the system focus better on the right image area, improving its accuracy compared to previous methods. Their method also uses several training tricks to keep learning stable and balanced.
What this means in practice
- •For autonomous vehicle teams: Improve vehicles' ability to locate places or objects described by passengers without detailed labeled data.
- •For medical imaging developers: Enhance tools that interpret verbal or written instructions to identify specific regions in medical images without extensive annotation.
Authors
Yu Yu Chang, Tzu-Hsuan Weng, Youssef Mohamed Amin, Mohaned Ahmed Elsayed, Hung-Hsuan Chen, Min-Te Sun, Sherine Shawky
Abstract
Referring Expression Comprehension (REC) localizes the image region described by a natural language expression, underpinning applications such as autonomous navigation, human-robot interaction, and medical image analysis. Under weak supervision, REC is challenged by the visual-linguistic semantic gap and the difficulty of inferring region-text correspondences from coarse image-level signals alone. Moreover, no single visual encoder captures all complementary properties---semantic alignment, fine-grained texture, and boundary awareness---while naively fusing multiple pretrained encoders yields heterogeneous, hard-to-reconcile representations. We propose LaMER, a weakly supervised REC framework that dynamically aggregates four frozen expert encoders through a language-conditioned Cross-Modal Routing Fusion module, adapting expert selection to each referring expression without instance-level bounding-box supervision. A Cell-Text Matching Head with Negative Anchor Augmentation, trained with a multi-objective loss combining contrastive, reconstruction, alignment, and load-balancing terms, further stabilizes optimization. LaMER achieves 77.67\%, 62.81\%, and 69.49\% Acc@IoU\(\geq 0.5\) on RefCOCO, RefCOCO+, and RefCOCOg, surpassing DViN and WeakMCN by up to 16.49 and 14.49 points, with ablations validating each component.