Simulation errors often cause robot benchmark failures not policies
When the Simulator Fails, Not the Policy: Auditing Contact Artifacts in Simulated Robot Benchmarks
Robotics
Summary
Robot success scores in simulations count every failure as the robot's fault, but this research shows many failures come from simulation bugs. The authors found that objects can be ejected wrongly due to how the physics engine detects collisions at small distances. They fixed a key setting in the simulator that stopped these artificial failures and improved accuracy of robot task success rates. Their auditing tool helps identify when simulation mistakes, not robot actions, cause problems in tests.
What this means in practice
- •For robotics simulation engineers: Detect and fix collision-related artifacts in robot simulation benchmarks to improve evaluation accuracy.
- •For robotics system developers: Re-run robot policy tests on fixed physics engines or report success ranges accounting for simulator artifacts.
Authors
Alex Zhou, Junhao Wu, Xueyang Zhou, Yixiao Huo, Hongcheng Gao, Di Zhu, Zhenghao Zhang, Yongchao Chen
Abstract
Simulation benchmarks score robot policies by success rate, and every failed episode counts against the policy. Many of these failures are the simulator's fault, not the policy's; we find the cause and fix it. Testing a vision language action model and a world action model in simulation, we find that the policy often places the object and the simulator then ejects it, and that the benchmark's own scripted demonstrations fail the same way. We present SimuGuard, an auditing plugin for LIBERO, RoboTwin, RoboCasa and RoboDojo. It leaves the evaluation unchanged. At every physics step, it records the state and flags an object that starts moving at a contact faster than the robot's contact or gravity account for. It replays a recorded episode bit for bit, and it reruns the episode with one simulation parameter changed. With it we trace these episodes on RoboTwin to the collision detection of the physics engine: for geometry one to two centimeters apart, the engine reports a penetration of about two centimeters, and the solver throws the object out at several meters per second. The cause is one constant: the distance algorithm discards triangles below a fixed area threshold, which valid triangles fall below at the centimeter scale. With that constant changed and nothing else, episodes with an artifact fall from 12 in 100 to none, and the success rate on five container tasks rises from 89% to 98%. Success rates in simulation thus count failures that the simulator causes. Benchmarks on PhysX 5.3.1 or earlier should rerun in a repaired engine. Where the engine cannot change, they should report the success rate together with the range it spans when every flagged episode fails or passes: 75.8% to 98.5% around a reported 83.1% for LingBot-VA.