RoboSemanticBench: Diagnosing Semantic Grounding in Action Prediction for VLA Models
2026-06-01 • Robotics
Robotics
AI summaryⓘ
The authors introduce RoboSemanticBench (RSB), a test to see if robots can understand complex instructions and pick the correct objects to act on. In this test, robots answer multiple-choice questions by grabbing the block with the right answer. They found that while robots can physically pick up blocks, they often fail to choose the correct one based on understanding the question's meaning. This shows a gap between what language-vision models know and how well they use that knowledge to guide robot actions.
vision-language modelsrobot action predictionsemantic groundingimitation learningembodied benchmarkmultiple-choice taskssemantic competencerobot manipulation
Authors
Bin Yu, Yao Zhang, Haishan Liu, Shijie Lian, Yuliang Wei, Xiaopeng Lin, Zhaolong Shen, Changti Wu, Ruina Hu, Bailing Wang, Cong Huang, Kai Chen
Abstract
Vision-language-action (VLA) models are built on the premise that semantic understanding from pretrained language or vision-language backbones should guide robot action prediction. Yet robot fine-tuning is optimized as imitation over task-specific action distributions, and many evaluations can be solved through visual or instruction-action shortcuts. We introduce RoboSemanticBench (RSB), an embodied benchmark for diagnosing semantic grounding in action prediction: whether post-trained VLA models can use complex instruction semantics to select and manipulate the correct physical target. In each episode, a robot receives a multiple-choice math or general-knowledge question, observes candidate answer blocks, and must grasp the block corresponding to the correct answer. RSB covers controlled arithmetic, grade-school mathematical understanding, and commonsense or factual understanding under four-choice and ten-choice suites. Across representative VLA models, we find that many policies learn to grasp candidate blocks but select the semantically correct block at near-random or below-random rates after controlling for grasp success, revealing a persistent gap between backbone-level semantic competence and action prediction.