Frontier agents show promise and limits in robot control tasks

AgentRoboCritic: Understanding the Capability of Frontier Agents on Direct Robot Control

RoboticsComputation and LanguageComputer Vision and Pattern Recognition

Summary

Controlling robots directly with language-driven AI agents is a challenging task. The authors developed a system called AgentRoboCritic that helps understand how these AI agents succeed or fail by checking each action against its intended goal. Their analysis shows that such agents can often recover from mistakes and complete tasks without extra training, but struggle with physical challenges like grasping objects precisely. They also created a new dataset and a tool to better detect why actions fail, making it easier to improve robot control.

What this means in practice

  • For robotics engineers: Diagnose and improve robot hand-eye coordination by analyzing action-level failures and recoveries in complex manipulation tasks.
  • For industrial automation teams: Use detailed action critiques to identify and address physical execution bottlenecks like grasping and alignment in robotic assembly lines.

Authors

Yiyang Du, Yuxuan Kuang, Qitao Zhao, Yixuan Wang, Zhanqiu Guo, Yi Xu, Xin Li, Wenbin He, Liu Ren, Chenyan Xiong

Abstract

Frontier multimodal language agents such as GPT-6 Astra have demonstrated strong capabilities in controlling robots directly via low-level numerical actions without high-level primitives. However, existing benchmarks report only episode-level success rates, which show neither how these agents succeed nor where they fail. In this paper, we present a systematic action-level analysis of frontier agents on direct robot control across 12 manipulation benchmarks. We introduce AgentRoboCritic, an intent-grounded critique framework in which the policy agent states a local intent alongside each action, and a critic evaluates the action's outcome and failure mode against this intent. Our action-level analysis demonstrates the promising capabilities of frontier agents on direct robot control, including closed-loop recovery from local execution failures, reliable semantic grounding across diverse scenes, and strong zero-shot manipulation without task-specific adaptation. It also identifies their current limitations, including task-relevant physical execution bottlenecks such as grasping and alignment, and uneven recovery across failure modes. Finally, we validate our critic on a 400-action human-annotated benchmark and release a dataset of 12,844 annotated actions alongside a fine-tuned open-source critic, Muse-Glimmer-30B, whose failure-mode accuracy rises from 32.58% to 74.24%, enabling low-cost action-level analysis.