World action models struggle with wording changes in spatial instructions
Same Goal, Different Words: Spatial Instruction Sensitivity in World-Action Models
Robotics
Summary
The study found that AI models designed to move objects based on spatial instructions often perform worse when the same goal is described using different words. The authors tested how consistent these models are when instructions are phrased from different viewpoints or with different wordings about object relations. They discovered that sometimes the models move the wrong object or misunderstand the instruction, even if the final spatial arrangement looks correct briefly. The work suggests that evaluating these models should include checking if instructions with the same meaning lead to the same actions and if the intended outcomes last over time.
What this means in practice
- •For robotic system developers: Improve robot control by testing if spatial instructions with different wording result in consistent actions to ensure reliable object placement.
- •For natural language interface designers: Design language interfaces that maintain consistent robot behavior despite variations in spatial instruction phrasing.
Authors
Ali-Adeeb Abbas, Esmaeil Seraj, Behrad Toghi, Taskin Padir
Abstract
World-action models should preserve the intended manipulation across equivalent descriptions of a spatial goal. We evaluate this consistency in 864 rollouts from three models across four RoboLab scenes, holding initial states and task goals fixed. Describing relations through the reference object rather than the object-to-move lowers the rate of reaching a stable placement by 12.2 percentage points on average. The effect of instruction rephrasing varies across spatial relations, so aggregate scores can hide sensitivity to wording. Standalone tests of the policies' language understanding identify relation reversals and show that adding images can disrupt otherwise correct text-only interpretations. Rollout analysis also distinguishes spatial placement from instruction following: moving the wrong object can satisfy the requested relation, and rates of stable placement fall from 43.2% at any time to 21.8% at episode end. These findings motivate a more thorough evaluation of world-action models' steerability, language understanding, and grounding. Evaluations should test consistency across equivalent instructions, verify that the intended object moved, and check whether the requested arrangement persists to the end of the rollout.