Robot software repair benchmark measures fixes by driving in simulation
RoboBugBench: Benchmarking Automated Repair of Real Robot Software Under an Audited Behavioral Oracle
Robotics
Summary
Robot software bugs show up as strange or wrong robot actions, but it’s hard to test fixes because robot code often lacks tests that check behavior. The authors created RoboBugBench, a set of real robot program bugs tested by running the robot in a simulator to see if the fix works. They found current AI and language models fix about a quarter to a third of the bugs, but it’s tricky to tell if a patch truly repairs the problem or just hides it. Their benchmark includes tools to catch fake fixes and highlights the challenges in observing robot behavior reliably.
What this means in practice
- •For robotics software engineers: Evaluate and improve automated bug fixes by verifying robot behavior changes in simulation rather than code tests alone.
- •For autonomous vehicle developers: Use simulation-driven benchmarks to better assess software patches that impact vehicle navigation and safety functions.
Authors
Alexey Tuzikov
Abstract
Language agents increasingly modify robot software, but whether they can repair it is unmeasured, because the domain supplies no oracle to inherit: a robot bug manifests as behavior, and robot repositories ship almost no tests that capture it. We present RoboBugBench, a benchmark of 33 real, execution-validated defects in a pinned ROS 2 navigation stack, in which a candidate patch is graded by compiling it and driving the robot through a physics simulation rather than by running a unit test. Three frontier models, each given 1 attempt per instance, repair 24.2%-36.4%, with no pairwise difference significant; difficulty tracks the oracle rather than the domain, at 66.7% on instances graded by process survival against 22.2% on those graded by what the robot did (instance-level p = 0.0027). Because the oracle is constructed rather than inherited we audit it adversarially, and report the decomposition that audit yields: counterfeit patches which suppress a failure without repairing the defect are accepted by the shipped rule 11/11, while a distributional rule over the same data rejects 4 at no cost in false rejects. The residual is an observability bound (what the observation channel discards, no predicate over it can recover), so discrimination must be bought with instrumentation rather than stricter criteria, and our repair rates are bounds: inflated by what the oracle cannot distinguish, deflated by unanimity's rejection of correct fixes under run-to-run noise. We release the corpus, criterion batteries, counterfeit suite and cached telemetry.