ActGaze improves robotic gaze for better precision in manipulation
ActGaze: Learning Action-Grounded Gaze through Counterfactual Visual Interventions for High-Precision Manipulation
Robotics
Summary
Robots often struggle with tasks that need very precise movements because their 'attention' tends to be spread over unimportant parts of what they see. The authors created ActGaze, a training method that teaches robots to focus their gaze on important parts of the scene related to the task, similar to how humans do. Instead of using extra instructions about where the robot should look, ActGaze figures this out by testing how changing parts of the visual scene affects the robot's action decisions. This method helped robots perform better on precise tasks by making them look at what really matters.
What this means in practice
- •For robotics engineers: Improve robot arm control by guiding visual focus to critical task areas for higher-precision manipulation.
- •For manufacturing automation teams: Enhance assembly line robots' accuracy by refining their visual attention to relevant components during complex tasks.
Authors
Jinxuan Zhu, Jiaheng Wang, Chao Tang, Mengfan Wang, Hao Wei, Shengbao Li, Hong Yin, Yiwen Gao, Chenrui Tie, Tingguang Li
Abstract
Current Vision-Language-Action (VLA) models often struggle with high-precision robotic manipulation. We attribute this limitation primarily to their visual attention being dispersed across task-irrelevant regions. To address this issue, we propose ActGaze, a training approach that guides VLA policies to gaze on task-relevant regions, much like humans gaze on critical visual cues while executing precise movements. Unlike prior methods that rely on external labels for gaze supervision, ActGaze derives spatial supervision directly from the VLA's own action objective by using counterfactual visual interventions to identify regions that are critical for action prediction. Extensive real-robot experiments on four high-precision robotic manipulation tasks demonstrate that ActGaze induces more focused visual attention on task-relevant regions and consistently outperforms the base VLA policy and other visual-grounding approaches.