Vision language action models vary in sensitivity to instruction changes
Do Vision-Language-Action Models Understand Instructions? A Mechanistic Interpretability Study on Language Grounding
Robotics
Summary
Some AI models that combine vision and language to perform tasks can still rely mostly on visual clues, even if they are given different or unclear instructions. The authors studied two such models and found that they don’t always pay much attention to changes in the task descriptions unless the instructions are empty or involve directions. Different parts inside each model are affected when instructions change, showing that some models handle instructions more deeply than others. This helps us understand how these models process language and when they truly rely on instructions versus visual signals.
What this means in practice
- •For robotic system developers: Improve instruction-following components by targeting model layers sensitive to directional language to build more reliable robot task performers.
- •For ai safety engineers: Assess and debug how vision-language models use instructions versus visual cues to reduce unpredictable behavior in deployed systems.
Tested on one dataset.
Authors
Theodor Wulff, Angelo Cangelosi
Abstract
Vision-Language-Action models are designed to generalise across environments and task descriptions, raising the question of whether their action generation actually depends on the language instruction, or whether they largely rely on visual cues and superficial correlations. Robustness to variance in the visual and linguistic observation space is critical for real-world deployment, yet VLAs lack explicit grounding modules and instead rely on the intrinsic language grounding capabilities of their Vision-Language model backbones. For this reason, we conduct a controlled mechanistic interpretability study on the language grounding capabilities of two state-of-the-art Vision-Language-Action models, $π_{0.5}$ and GR00T N1.7, by applying activation and attribution patching to the residual stream of the action generation modules. We systematically corrupt the task instruction of input samples of the LIBERO benchmark following five strategies: synonym replacement, semantic scaling, directional corruption, random object substitution, and empty string. Our experiments find that both models are comparatively insensitive to abstract rephrasing and to referencing non-existent objects, but react strongly to empty task descriptions and, especially, to directional language. During action generation, this sensitivity is concentrated in different loci for each model: mainly in the early, periodic cross-attention layers for GR00T N1.7, versus distributed across the earliest and selected later layers for $π_{0.5}$. For GR00T N1.7, directional perturbations drive some of the largest causal effects while leaving the internal representational geometry comparatively unchanged, a dissociation we do not observe clearly for $π_{0.5}$. Finally, the reliability of attribution patching is model-dependent: it closely tracks activation patching for GR00T N1.7 but not for $π_{0.5}$.