InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation
2026-08-24 • Robotics
Robotics
AI summaryⓘ
The authors created a new test called InstructMove to better check if robot models truly understand instructions given in natural language when moving objects. Unlike older tests where the right choice was obvious, InstructMove makes sure multiple moves seem possible visually, but only one matches the language instruction. This test breaks down following instructions into steps like recognizing object categories, attributes, and spatial reasoning. Their experiments show InstructMove can reveal when robots rely too much on visual cues and that training with this data helps robots perform better with real-world instruction-following tasks.
vision-language-action (VLA) modelsrobot manipulationnatural-language instructionsinstruction-followingpick-and-placesemantic distractorscategory identificationattribute discriminationspatial reasoningcompositional tasks
Authors
Mengao Zhao, Ziang Li, Chaodong Huang, Mengchen Ma, Haoyi Jiang, Yiwei Jin, Xinjie Wang, Yun Du, Xuewu Lin, Taojun Ding, Hongyu Xie, Jackson Jiang, Chunlei Yu, Kaihua Zhang, Lichao Huang, Liu Liu, Tianwei Lin, Zhizhong Su
Abstract
Vision-language-action (VLA) models have made general-purpose robot manipulation increasingly plausible by conditioning robot actions on natural-language instructions. A key test of such generality is whether policies actually follow language instructions. Yet many manipulation benchmarks leave this ability underdetermined: the intended object or destination is often visually salient or uniquely feasible, allowing policies to succeed without grounding the instruction. We argue that instruction-following evaluation should be text-indispensable: multiple actions should be visually and physically plausible, while only one should be consistent with the language instruction. We introduce InstructMove, a text-indispensable benchmark for instruction-following manipulation. InstructMove instantiates this principle in pick-and-place scenes with semantic distractors, decomposing instruction following into category identification, attribute discrimination, spatial reasoning, and compositional pick-and-place. InstructMove supports a train-eval protocol with InstructMove training data and held-out evaluation tasks, with additional diagnostics for language dependence. Experiments with representative VLA policies show that InstructMove provides a controlled testbed for diagnosing visual shortcuts and that InstructMove simulation data can improve real-world instruction-following manipulation performance. Code: https://github.com/HorizonRobotics/RoboOrchardSim