General purpose models tested on robot manipulation tasks
CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation
RoboticsArtificial Intelligence
Summary
It is hard for AI models to understand what they see and then use robotic arms to move or manipulate objects correctly. The authors designed a test called CodeActionBench with 25 different robot tasks to see how well AI can do this by writing and running code as instructions. Their best AI setup solved most tasks sometimes, but there were still problems like aligning objects or knowing when a task was finished. This test helps measure how good AI is at turning what it understands into actual robot actions and shows common mistakes.
What this means in practice
- •For robotics engineers: Evaluate and improve robot control algorithms by comparing agentic code-based policies on a standard set of manipulation tasks.
- •For automation integrators: Test AI-based code generation methods for robotic manipulation in diverse scenarios without specialized hardware or software modules.
Authors
Yiheng Lyu, Xueying Jiang, Wenhao Li, Shijian Lu, Gongjie Zhang
Abstract
How well can general-purpose multimodal models turn visual understanding and reasoning into embodied manipulation via executable code? We introduce CodeActionBench, a benchmark of 25 manipulation tasks that evaluates this capability through agentic Code-as-Policy. Without task-specific fine-tuning, demonstrations, external specialist perception or grasp modules, privileged scene state, or predefined task policies, agents should select visual evidence, form task-relevant 3D estimates, construct manipulation targets, and iteratively execute and revise their policies. A shared robot API provides RGB observations, calibrated geometric operations, robot feedback, and bounded motion, leaving task-dependent decisions to the evaluated agent. Fixed task instances, resource budgets, and a hidden physical-outcome verifier support controlled comparisons across models and harness configurations. Extensive evaluations across nine configurations and 675 attempts achieve success rates ranging from 2.7% to 73.3%. The strongest configuration, GPT-6 Astra with Codex CLI, solves 22 of 25 tasks at least once in three attempts, demonstrating the best performance while still leaving substantial room for improvement. Trajectory analyses reveal difficulties in spatial alignment, object retention, and completion judgment, including task failures despite successfully completed motions. CodeActionBench provides a controlled testbed for measuring how general-purpose models translate their capabilities into manipulation behavior and for examining typical failure scenarios in that process.