Generalized cycle method improves region understanding and localization
Generalized CycleGRPO for Fine-Grained Multimodal Understanding and Grounding
Computer Vision and Pattern Recognition
Summary
Understanding and finding specific parts of images or videos based on language is hard because models usually treat description and localization separately. The authors introduce Generalized CycleGRPO, a method that links describing a region and locating it back in images or videos using a feedback loop. This approach helps the model check its own accuracy without extra labeled data. They also create a test called DOVE to verify how well the method distinguishes unique events in videos. Experiments show their method improves accuracy over previous ones on multiple benchmarks.
What this means in practice
- •For computer vision teams: Improve models that identify and describe specific regions in images or videos with better accuracy using a unified learning approach.
- •For video analytics developers: Enhance event detection systems to distinguish and localize similar recurring events in videos more precisely without relying on extra labeled data.
Authors
Xin Zhang, Haochen Wang, Beibei Lin, Zhaoxiang Zhang, Robby T. Tan, Jason Li, Dacheng Tao
Abstract
Fine-grained multimodal intelligence requires models to both understand a designated region and localize it from language. Existing methods learn these directions with separate objectives and task-specific annotations. We present Generalized CycleGRPO, the first unified reinforcement learning framework coupling region understanding and localization across spatial, temporal, and spatio-temporal domains. Given a region, a multimodal large language model generates candidate descriptions and then localizes each description to reconstruct the input region. Reconstruction overlap provides a verifiable reward for captioning, while individual localization rewards improve the inverse mapping. A common region-space formulation extends this cycle from image masks and boxes to video intervals and timestamped spatial evidence, without reference captions or an external language-model judge during reinforcement learning. In videos, the cycle encourages descriptions that distinguish a target event from similar recurrences. We introduce DOVE (Distinctive Occurrence-level Video-caption Evaluation), a human-verified contrastive benchmark for this temporal distinctiveness. CycleGRPO improves SAMTok's overall gIoU from 57.5% to 68.2% on GroundingSuite and average gIoU from 71.2% to 78.7% on GRES. For temporal tasks, it achieves 51.9% tIoU on Charades and 33.3% Temporal Evidence Coverage on DOVE, compared with 45.6% and 14.8% for the Qwen3-VL initialization. These results establish region-text cycle consistency as a general self-evaluation principle for multimodal understanding and localization. Models, code, and DOVE are available at https://devinxzhang.github.io/Generalized-CycleGRPO-Page/.