Frontier AI models show mixed success on robot manipulation tasks
From Reasoning to Action: Benchmarking Frontier AI Models on LIBERO-10 Robot Manipulation Tasks
RoboticsArtificial Intelligence
Summary
Controlling robots to pick up and arrange objects is a tough challenge. This work tests six advanced AI models to see how well they can guide a robot's movements using only visual input and basic commands, without special training for each task. The best model succeeded about 44% of the time, meaning there is some potential but also many difficulties in translating smart thinking into precise robot actions. The researchers also created a new framework to fairly evaluate and compare such AI-driven robot control approaches.
What this means in practice
- •For robotics engineers: Test AI models for grasping and placing objects without specialized training using a standardized evaluation framework.
- •For automation integrators: Assess AI control strategies for automating complex multi-step manipulation tasks in industrial settings.
Authors
May Espinola, Qifan Zhang, Yu Xiang
Abstract
Recent advances in frontier AI models have demonstrated remarkable capabilities in visual understanding, reasoning, planning, and tool use, raising the possibility of using general-purpose AI models for robot manipulation without task-specific training. However, their manipulation capabilities have not been systematically evaluated on standardized robotics benchmarks. In this report, we present a unified closed-loop manipulation framework for evaluating frontier AI models on the LIBERO-10 benchmark. Our framework enables models to control a robot directly through visual observations, robot proprioception, and structured tool calls, primarily by specifying target end-effector poses and gripper commands, without task-specific fine-tuning or pretrained manipulation skills. We evaluate six frontier AI models, including GPT-6 Astra, GPT-6.1 Sol, GPT-6 Luna, Claude Opus 5, Claude Opus 5.5, and Gemini-4 Argon, across three reasoning effort levels, resulting in 18 configurations and 1,800 evaluation episodes. Our results show that GPT-6 Astra achieves the highest success rate of 44%, followed by Gemini-4 Argon at 43%. Increasing reasoning effort generally improves manipulation performance, but the benefits vary across models and come at additional inference cost. We also observe substantial differences across tasks, with models performing relatively well on object placement tasks but struggling with precise spatial arrangements and complex multi-step interactions. These findings demonstrate both the potential and limitations of frontier AI models for robot manipulation, highlighting the challenges of translating reasoning capabilities into reliable physical actions. We release our evaluation framework and experimental results to support reproducible benchmarking and future research on reasoning-based robot manipulation.