InstanceBench benchmarks reasoning in object segmentation from language
InstanceBench: Diagnosing Referential Reasoning and Target Identity in Referring Expression Segmentation
Computer Vision and Pattern Recognition
Summary
Linking words people use to describe objects with precise outlines of those objects in images is hard because many objects may look similar. The authors created InstanceBench, a large collection of images and carefully tested descriptions that check if AI models not only find the right object but also understand different ways to describe it. Their tests reveal that current AI models often pick the wrong object rather than making mistakes drawing its outline. They also show that giving AI more targeted training helps improve this skill. This benchmark helps find exactly where AI systems struggle and guides improvements.
What this means in practice
- •For computer vision teams: Evaluate segmentation models for understanding detailed natural language references and improve object identification accuracy under complex descriptions.
- •For natural language processing teams: Diagnose language grounding abilities in vision-language models to refine how models connect words to specific image parts.
Authors
Yuchen Li, Shaoyang Zhou, Yiran Wang, Ruiyi Deng, Haoyu Wang, Ziru Wei, Zhen Zhao, Luping Zhou
Abstract
Referring Expression Segmentation (RES) links natural-language descriptions to pixel-level object masks. Yet standard evaluation provides limited insight into instance-level referential reasoning: it does not systematically distinguish referential logics, test target preservation across valid grounding paths, or separate target-selection from mask-generation errors. We introduce InstanceBench, an instance-centered diagnostic benchmark comprising 6,194 images, 9,264 target instances, and 25,077 human-verified expressions. Each target-centric expression set (TCES) fixes the image and target mask while pairing a minimal expression with a same-target variant that uses another valid cue or grounding path. A compact referential-logic taxonomy spans direct target evidence, same-class selection, relational and compositional grounding, and exclusion, while logic-critical construction suppresses simpler shortcuts. Identity-aware metrics measure target retention and set-level success while separating selection from mask-generation errors. Across 22 native-mask RES checkpoints from 18 model families, the strongest checkpoint reaches 67.1% mIoU but only 59.6% All@0.7. Controlled interventions confirm language sensitivity, while failure decomposition identifies target selection rather than mask decoding as the main bottleneck. On a controlled training subset, matched supervision improves identity-aware performance, showing that the diagnosed capability responds to targeted supervision. Collectively, InstanceBench supports a measure-diagnose-improve cycle: measuring target consistency across grounding paths, localizing failure sources, and evaluating targeted interventions.