Embodied agents struggle to link instruction meaning with actions
STAGE: Diagnosing Semantic Transfer at Grounded Execution in Embodied Agents
RoboticsComputation and Language
Summary
Understanding a command is not enough for a robot to act correctly; the meaning must control what the robot does. The authors show that current systems can often identify the correct object in instructions but fail to translate that understanding into appropriate actions. They developed a test (SAT-Bench) that keeps the scene and robot state fixed but changes the instruction meaning to diagnose this problem. Their solution, VISA, helps convert understanding into better decisions during execution, reducing mistakes drastically without retraining the main system.
What this means in practice
- •For robotics software developers: Improve robotic systems by integrating execution-time checks that link instruction meaning directly to robot actions, reducing erroneous behaviors.
- •For autonomous vehicle engineers: Enhance instruction-following modules in autonomous vehicles by diagnosing and fixing mismatches between command understanding and action execution.
Authors
Baosheng Jin, Yushen Liang, Hua Shen
Abstract
Embodied language grounding requires more than identifying the referent of an instruction: recovered semantics must also control the action an agent exposes. We study this missing link as a semantic-action gap, where instruction semantics are recoverable but weakly expressed in native continuous actions. We introduce SAT-Bench, a fixed-observation counterfactual benchmark that holds the visual scene and agent state fixed while changing only instruction semantics. On LIBERO target-name and pixel-grounded relation swaps, target recovery reaches 100.0% and 95.8%, whereas OpenVLA action sensitivity remains only 6.8% and 7.7%. The gap persists across 1,000 additional compositional and temporal/procedural counterfactuals, with overall action sensitivity of 6.1%. Hidden-state, threshold-free, cross-policy, and rollout diagnostics further support this semantic-action transfer failure. We introduce VISA, a lightweight execution-time interface that converts recovered semantics into ALLOW, DEFER, target-consistency, and verified-selection decisions. VISA reduces invalid-instruction blind execution from 92.7% to 2.8% while preserving 94.0% of normal commands, and verified selection further improves target-consistent action exposure without updating the underlying policy. Overall, embodied language evaluation should measure semantic-action transfer, not semantic parsing alone.