Embodied agents struggle to truly follow instructions in complex tasks
RoboFollow: Unveiling the Instruction Following Mirage in Embodied Agents
Robotics
Summary
Robots that move around and complete tasks often seem very good at following instructions, but this can be misleading. The authors found that in many cases, the robot's environment only allows one possible task, so it doesn't really need to understand the instructions to succeed. They built a new test called RoboFollow that makes tasks more varied and complex, forcing robots to really use the instructions to decide what to do. Their experiments show that many current robot systems don't actually understand instructions well when the task is harder or more detailed.
What this means in practice
- •For robotics engineers: Evaluate and improve robot instruction comprehension by testing on varied task scenarios demanding genuine language understanding.
- •For machine learning developers: Design better vision-language models capable of handling complex spatial and logical instructions under varied scene configurations.
Authors
Chang Guo, Yukun Xie, Bohan Tan, Zheng Chang, Zhaokai Yin, Qianli Ma, Yingqiao Wang, Chao Liang, Zhipeng Zhang
Abstract
Modern embodied agents achieve impressive success rates, yet their actual instruction-following ability is far weaker than these numbers suggest. We trace this illusion to a structural property we term low scene entropy: when a visual scene admits only one valid task, language becomes redundant and a policy can score highly while barely using it. We introduce RoboFollow, a diagnostic benchmark with three principles: (1) High Scene Entropy: each training scene supports multiple kinematically distinct task branches, making vision alone insufficient and forcing reliance on language. (2) Hierarchical Diagnostic Protocol: a four-level protocol (L0--L3) progressively perturbs visual layout and semantics, probing whether equivalent instructions yield consistent behavior and distinct ones yield discriminable behavior across spatial relations, attributes, trajectory constraints, and logic. (3) Confound-Controlled Diagnosis: we simplify interaction objects, restrict actions to the trained repertoire and report stage-wise Intent and Execution scores, isolating comprehension from motor execution. Evaluation of nine VLA and WAM policies shows that strong L0 performance, where attained, does not reliably transfer to L1--L3 under our fine-tuning setup. Representative mitigations, including stronger VLM backbones, QA co-training, LangForce, and Classifier-Free Guidance, all fail to close this gap. RoboFollow exposes genuine instruction following as a critical, overlooked bottleneck. Code and dataset are available at https://github.com/AutoLab-SAI-SJTU/RoboFollow and https://huggingface.co/datasets/AutoLab-SJTU/robofollow-data.