Embodied-BenchForge builds better robot task benchmarks with error checking
Embodied-BenchForge: A Closed-Loop Agentic Workflow for Embodied Benchmark Construction
Artificial Intelligence
Summary
Creating complex tests for robots and AI that interact with the world is hard because mistakes in early steps can cause big problems later. The authors present Embodied-BenchForge, a system that not only builds these robot task tests but also double-checks and fixes errors along the way. It uses a clear process where tasks are made, checked, and repaired to ensure high-quality benchmarks that help better evaluate robot and AI abilities. They showed this system can create several diverse test sets and improve reliability compared to previous methods.
What this means in practice
- •For robotics developers: Generate reliable and diverse task sets to evaluate robot navigation and manipulation in multiple scenarios with built-in error checks.
- •For ai benchmarking teams: Create large-scale embodied AI benchmarks that combine task synthesis and automated repair to improve dataset quality and reuse across evaluations.
Authors
Baoyang Jiang, Fengchun Zhang, Leyuan Wang, Haotian Li, Yida Wang, Zhe Ji, Jinshan Lai, Xi Ren, Danyang Li, Zheng Yang, Jianwei Hu, Qiang Ma
Abstract
Agentic systems offer a promising way to automate embodied benchmark construction, but existing approaches typically cover isolated stages or remain specialized to predefined environments and task families. More importantly, multi-step construction produces dependent intermediate artifacts that are often passed downstream without artifact-specific verification, allowing local defects to propagate into the final benchmark. We present Embodied-BenchForge, an agentic framework that transforms user-specified evaluation intents into complete embodied benchmark artifacts. It formulates construction as Closed-Loop Benchmark Synthesis, integrating forward artifact synthesis with backward verification and repair. Skill-Orchestrated Artifact Synthesis composes typed and reusable skills into executable workflows, while an artifact dependency graph records intermediate outputs and their dependencies. Requirement-Guided Verification and Repair applies artifact-specific contracts throughout construction and uses provenance to trigger local re-execution or upstream rollback when verification fails. Embodied-BenchForge constructs six benchmarks covering diverse embodied scenarios in the Offline EQA Track, together with one interactive benchmark containing 220 executable tasks in the Interactive Embodied Track. Evaluations of representative MLLMs and embodied agents show that the benchmarks distinguish model capabilities in both observation-based understanding and closed-loop execution. Quality assessment and ablations validate benchmark quality and the effectiveness of verification and repair, while repair and skill-reuse analyses demonstrate efficient localized recovery and cross-benchmark reusability.