ReferFlow improves visual grounding for complex language queries
ReferFlow: Beyond Final-Target Localization for Complex Referring Expressions
Computer Vision and Pattern Recognition
Summary
Sometimes computers have trouble understanding complicated descriptions to find objects in pictures because these descriptions refer to several parts step-by-step before the final object. The authors created a new way to test and improve how well models can follow these complex instructions by breaking down the problem into smaller steps called trajectories. They designed a model called ReferFlow that remembers earlier parts of the description to better identify intermediate objects and the final target. This approach helps computers do better at tasks that require understanding multiple related objects in images.
What this means in practice
- •For computer vision engineers: Enable models to accurately identify and track multiple related objects in images from complex natural language instructions.
- •For interactive ai developers: Improve AI assistants or robots that interpret multi-step user commands involving sequential object references in visual scenes.
Authors
Tianyi Gao, Han Fang, Xin Wei, Yaoyao Zhong, Zhaoyan Pan, Weijie Zhu, Hongbo Sun, Zhiyu Lin, Kaicheng Yang, Jinglin Xu, Ye Yuan, Hao Li, Hao Sun, Jingmin Xin
Abstract
Multimodal Large Language Models have made significant progress in referring expression comprehension (REC), yet most REC benchmarks still focus on final-target localization. Complex queries often involve intermediate visual references with different dependencies on prior grounding, but final-only evaluation leaves this process unexamined. To address this gap, we formulate trajectory-level REC, in which models receive an ordered sequence of grounding steps constructed from a single complex query and localize the target regions associated with each step. We instantiate this setting in ReferFlow-Bench, which characterizes structured region transitions through four trajectory types, namely Refinement, Shift, Reset, and Mixture. Experiments show that current models struggle to consistently ground intermediate visual references across successive steps. We further propose ReferFlow, which decouples interleaved text-latent trajectory generation from coordinate decoding. Specifically, Stateful Region Latent Reasoning represents each target region as an object-indexed state composed of continuous latents, with explicit visual and spatial supervision to preserve region information for subsequent grounding. Trajectory-Aware Parallel Decoding then recovers all coordinate blocks in parallel through four geometry-constrained refinement rounds, with each block conditioned on its corresponding trajectory prefix, thereby reducing within-box sequential dependencies and eliminating cross-box autoregression. We further apply Grounding-Aware GRPO to promote consistent grounding across steps and learn adaptive trajectory generation from complex REC queries. ReferFlow achieves leading performance on general REC benchmarks and ReferFlow-Bench.