Vision language action models struggle to follow how instructions
Do VLAs Really Follow Instructions? Diagnosing How-Level Instruction Following in Vision-Language-Action Models
Computer Vision and Pattern RecognitionRobotics
Summary
Many AI systems that combine vision and language are expected to follow detailed instructions on how to perform tasks. This paper finds that current models do not improve when given step-by-step (how-level) instructions and often perform worse. The authors suggest that training methods limit the models' ability to respond to such instructions, even when attempts are made to improve it. They introduce a new test that better evaluates following how-level directions, showing that current models still fail on harder or unexpected instructions. So, being able to follow detailed instructions remains a challenge for these systems.
What this means in practice
- •For robotics developers: Improve training methods to make robots respond better to detailed instructions for complex tasks.
- •For virtual assistant teams: Design better evaluation tests to measure how well assistants follow step-by-step user commands.
Authors
Sangwu Park, Wonjoong Kim, Yeonjun In, Chanyoung Park
Abstract
Vision-language-action (VLA) models are expected to follow instructions like their language backbones, yet prior work has focused on What-level instructions that specify the task rather than How-level instructions that specify how to carry it out. Through our diagnoses, we find that How-level instructions do not help current VLAs and lower task success. We hypothesize that pairing each training episode with a single instruction limits How-level instruction following, and show that training with segment-level enriched instructions makes actions respond to such instructions without consistently raising task success. To measure How-level instruction following beyond standard benchmarks, we introduce SIMPLER-IN, where success requires conditions given only in the instruction. It shows that the gained responsiveness does not extend to demanding or out-of-distribution How-level instructions. How-level instruction following thus remains an open challenge.