AI summaryⓘ
The authors focus on improving Text-Based Person Search, which means finding images of people using detailed written descriptions. They point out that current models, especially those using CLIP, miss fine details because they mainly learn from short captions and lack detailed region-level guidance. To fix this, the authors created ROGLE, a system that automatically matches parts of images with sentences to teach the model better fine-grained understanding without needing expensive manual labeling. They also built a new large dataset called P-VLG with many detailed annotations and longer captions to test their method more thoroughly. Their experiments show that ROGLE works better than existing methods, especially with complex queries.
Text-Based Person SearchCLIPfine-grained alignmentregion-level annotationsRegion-to-Sentence Matchingcontrastive learninglocal alignmentP-VLG Benchmarkimage-caption datasetsmulti-granular learning
Authors
Zequn Xie, Xibei Jia, Sihang Cai, Shulei Wang, Tao Jin
Abstract
Text-Based Person Search (TBPS) aims to retrieve pedestrian images using natural language queries. However, existing TBPS models, especially those based on CLIP, struggle with fine-grained understanding due to global representational bias and semantic sparsity inherited from training on short captions. This results in weak fine-grained alignment, exacerbated by the scarcity of region-level annotations. To address this, we propose ROGLE (Robust Global-Local Embedding), a unified framework that overcomes reliance on costly manual annotations through an automated Region-to-Sentence Matching (RSM) strategy. RSM automatically mines pseudo region-sentence pairs for scalable fine-grained supervision. Furthermore, ROGLE employs a multi-granular learning strategy that fuses global contrastive learning with region-level local alignment. We also introduce the P-VLG Benchmark, a large-scale dataset constructed by curating and enriching images from established public benchmarks. It features over 100,000 annotated regions and rich long-form captions, making it the first TBPS benchmark to support both global and local assessment protocols. Extensive experiments show that ROGLE significantly outperforms existing approaches, particularly on challenging long-form queries. Code and the P-VLG benchmark will be made publicly available.