Open-vocabulary models tested for robust robot visual understanding
Pro-Bench: Prompt-Robust Open-Vocabulary Visual Grounding Across Real-World Heterogeneous Environments
Computer Vision and Pattern RecognitionRobotics
Summary
Robots need to find objects by listening to natural language, but current tests mostly use simple labels and internet pictures that don’t reflect real-world challenges. The authors created Pro-Bench, a big new test that includes lots of photos from real robot settings like underground, industrial, and outdoor environments, with many detailed object descriptions. They tested various models and found that most work best with short simple labels, while free-form sentences often confuse them. This benchmark helps understand how well these models really work when robots face real-world, varied tasks.
What this means in practice
- •For robotics engineering teams: Evaluate and improve visual grounding models for robots operating in diverse real-world environments using Pro-Bench's challenge scenarios.
- •For industrial automation developers: Test how well open-vocabulary models recognize and localize objects from complex queries in industrial settings with Pro-Bench data.
Authors
Linus Nwankwo, Muslim Alaran, Christian Rauch, Stanley Chukwuebuka Obilikpa, Elmar Rueckert
Abstract
Open-vocabulary visual grounding enables robots to localise task-relevant entities from natural-language queries without dependence on predefined perceptual taxonomies. However, existing benchmarks largely rely on short category labels and web-scraped imagery, leaving it unclear whether open-vocabulary models can robustly ground diverse queries and visual conditions under real deployments. We introduce \textbf{Pro-Bench}, a prompt-conditioned benchmark for open-vocabulary visual grounding in heterogeneous, real-world environments. Pro-Bench includes $13k+$ RGB frames from independent robotic domains (subterranean, industrial, indoor, outdoor, urban), with $74.5k$ manual instance annotations and $515$ target queries covering categorical, attributive, relational, affordance, state, part-whole, negative, and compositional semantics. We benchmarked $16$ open-vocabulary model configurations in strict zero-shot inference, measuring localisation accuracy across IoU thresholds, end-to-end inference latency, prompt-induced performance variation, and target recovery consistency. Our results show that prompt-robustness is strongly architecture-dependent. Most model configurations ($10/16$) perform best with short category labels, whereas free-form queries yield the highest accuracy for only one. Moreover, similar aggregate mAP can conceal substantial differences in consistent target recovery across reformulations. Pro-Bench enables systematic evaluation of these gaps and supports prompt-robust visual grounding. Pro-Bench: https://pro-bench.github.io/.