Simulation benchmark improves robot manipulation evaluation accuracy
X2Real: an eXtensive simulation benchmark for real-world generalist policies
Robotics
Summary
Measuring how well robots can manipulate objects is tough because simulators often don’t match real-world conditions well and cover only a few tasks. The authors created X2Real, a system that closely mimics real robot behaviors and tests many different tasks in a way that is fair and reliable. They calibrated simulation settings to match real robot performance and included a wide range of skills, from simple moves to complex tasks like understanding language and using both hands. This helps build better robot programs by giving a trustworthy way to test them before using real robots.
What this means in practice
- •For robotics engineering teams: Evaluate robot control policies reliably with simulation closely matching real hardware performance across many task types.
- •For industrial automation developers: Test and refine robotic manipulation software in a simulated environment that faithfully reflects real-world conditions before deployment.
Authors
Lian Ruan, Jade Yang, Sherphylan Gao, Felix Gao, Kyson Liang, Galen Liu, Ligo Wu, Lane Jin, Guu Gu, Bevan Xie, Cloud Yan, Zongzi Yuan, Kino Luo, Emma Chen, Shuwen Chen, Yang Ping, Miles Guo, Rain Sun, Kayden Zhang, Alex Du, Ruihai Wu, Liang Hao, Zhaoshuo Li, Roy Gan, Hao Wang, Qian Wang
Abstract
Generalist robot manipulation policies have developed rapidly, yet their reliable evaluation remains challenging due to fundamental flaws in existing simulation benchmarks: prominent sim-to-real gaps, narrow task coverage, and unfair evaluation caused by ambiguous training-test pipelines. Prior works only partially resolve these issues and lack simultaneous faithfulness, diversity, and fairness, while static benchmark designs fail to sustain long-term policy development. We present X2Real, an evolvable simulation benchmark for faithfully evaluating the real-world performance of robotic manipulation policies based on Nvidia Isaac Lab-Arena. Following three core principles (faithfulness, diversity, and fairness), X2Real calibrates simulation visual and physical properties to align with real hardware, achieving a 0.84 linear correlation between simulated and real-robot evaluation results. It features a comprehensive taxonomy with 10 capability dimensions and 44 hierarchical long-horizon tasks, covering basic manipulation skills and advanced capacities such as visual grounding, language understanding, and bimanual control. We further adopt multi-axis domain randomization and strictly disjoint training-evaluation pipelines to mitigate benchmark exploitation and ensure credible evaluation. Powered by a custom physical domain-specific language, the Mana simulation ecosystem supports modular task design and iterative performance analysis, alongside a nearly 300-hour annotated simulation trajectory dataset. X2Real offers a faithful, diverse, and fair evolving evaluation infrastructure, effectively bridging the sim-to-real evaluation gap and supporting the advancement of generalist robotic manipulation policies.