Reasoning-Guided Part-Level Visual Grounding via Reinforcement Learning
Computer Vision and Pattern Recognition
Summary
The authors found that current multimodal large language models (MLLMs) struggle to identify parts of objects when given a language query because they try to find both whole objects and parts in one step. To fix this, they created a two-step method called Object-Part Hierarchical Reflective Grounding (OP-HRG) that first finds the whole object and then locates the specific part inside it. Their method also includes a self-check to verify results and can retrace steps if needed. When trained on a special framework, their model performs better at recognizing parts than larger existing models and works well on different datasets.
Authors
Kazi Sajeed Mehrab, Hani Alomari, Najibul Haque Sarker, Chia-Wei Tang, Zaber Ibn Abdul Hakim, Anuj Karpatne, Chris Thomas
Abstract
Multimodal large language models (MLLMs) ground whole objects well from free-form language queries, but they struggle when the query names a part rather than the object. We trace this to a missing object-part hierarchy, since parts are localized in the same single step used for objects. We propose Object-Part Hierarchical Reflective Grounding (OP-HRG), a coarse-to-fine reasoning-guided grounding strategy that first localizes the parent object and then the part within it. A self-check then reflects on the result, with an extension to re-encode the predicted crop to inspect the region it is correcting. We introduce a part-aware GRPO framework to train our pipeline with stage-wise rewards. A 4B model trained this way outperforms 7B grounding LLMs and SAM3 across PascalPart, PartImageNet, and InstructPart, and transfers to reasoning segmentation.