Vision language action models struggle with implicit object references
RoboIRGBench: Benchmarking Implicit Referential Grounding in Vision-Language-Action Models
RoboticsComputer Vision and Pattern Recognition
Summary
Robots often get instructions that mention objects clearly, but in real life, humans refer to things in less obvious ways, expecting the robot to figure it out from context. The authors created a new test called RoboIRG-Bench to see how well robots handle these tricky references. They found that robots that do well with clear instructions often fail when they have to guess what the human means based on context alone. This problem happens both in computer simulations and with a real robot arm, showing it’s an important challenge for building smarter robots.
What this means in practice
- •For robotic system developers: Improve robot software to handle instructions with indirect or implicit object references, enhancing real-world task performance.
- •For industrial automation teams: Deploy robots that better interpret less explicit human commands for flexible manufacturing and assembly tasks.
Authors
Aernaer Akelijiang, Jiannan Li, Zhineng Chen, Jingjing Chen, Bin Zhu
Abstract
Vision-Language-Action (VLA) models have shown strong capabilities in robotic manipulation, yet existing benchmarks typically assume that task-relevant information is explicitly specified in the instruction. In practice, however, humans frequently refer to objects, quantities, and relations implicitly, requiring robots to recover the intended target from linguistic and perceptual context. We study this capability as Implicit Referential Grounding (IRG) and introduce RoboIRG-Bench, a manipulation benchmark designed to systematically evaluate it. Built upon RoboMME, RoboIRG-Bench contains 40 variants derived from 11 tasks and covers four challenges, including direct, reasoning-mediated, spatial, and contextual referential grounding. As IRG often requires retaining and retrieving previously established context, we evaluate representative VLAs spanning different memory mechanisms. Our evaluation reveals a noticeable referential robustness gap. Models that perform well under explicit instructions can degrade sharply when the same task-relevant information must be recovered from context. Reasoning-mediated and spatial references are particularly challenging, while models using external VLMs show greater robustness but still exhibit significant failures. Moreover, replacing the external VLM with a stronger model does not eliminate these gaps. We further validate these findings on a Franka Research 3 robot arm, where the gap persists under real-world manipulation and manifests as both incorrect referent grounding and downstream execution failures. These results establish IRG as a distinct and underexplored capability for reliable robotic instruction following and highlight the need for VLAs that can robustly integrate language, perception, reasoning, and action.