Robot software issue fixes validated by running tests across multiple domains
Robo-SWE: An Executable Issue-Resolution Dataset Mined from Robot Software Repositories
Software EngineeringRobotics
Summary
Fixing problems in robot software is hard because robots rely on special tools and hardware that are tricky to test. The authors collected a large set of real robot software issues along with the fixes and tests that show the fixes work when run. They tested several coding agents on these robot-related tasks to see how well the agents could solve the issues. This dataset can help improve how robot software is tested and how automated tools fix problems in this specialized area.
What this means in practice
- •For robot software developers: Use the dataset to test and improve automated tools that fix coding issues in robot software packages including ROS 2 and simulation frameworks.
- •For ai coding tool providers: Train and benchmark coding assistants on realistic robot software problems that require domain-specific knowledge and executable validation.
Authors
Vahid Haratian, Vahid Ansari, Mohammadhossein Malmir, Alois Knoll
Abstract
Coding agents are evaluated mostly on general-purpose software, but domain-specific systems require domain-aware evaluation. Robot software is a particularly challenging case: Robot Operating System 2 (ROS 2) packages have their own build tools, simulators and learning frameworks are built on physics engines, and behavior that depends on timing and hardware is hard to test. We present Robo-SWE, a dataset of 236 executable issue-resolution tasks mined from merged pull requests in 28 public robotics repositories in four domains: ROS 2 systems, simulation and physics engines, robotics algorithm libraries, and robot-learning frameworks. Each task contains the original issue text, a container recipe for the repository at the commit before the fix, the developers' fix, and the developers' tests, which fail before the fix and pass after it in repeated offline runs. The dataset also keeps a provenance ledger of all 1,567 candidate pull requests, with the stage each reached and the reason for every rejection, and, as a first use, 3,540 graded attempts by five coding-agent setups with recorded patches and 3,529 available trajectories. Across three runs per primary setup, resolve rates range from 45.8% for Claude Sonnet 5.5 to 56.5% for Claude Opus 5.5; GPT-6.1 Sol reaches 54.1% at maximum effort and 52.5% at medium effort, and GPT-6 Astra 51.4%. GLM-5.3 resolves 46.2%, and Kimi K3.1 Pro 43.9%. The tasks are validated by execution; verification of their fairness by human reviewers is future work. The dataset and public leaderboard are available online.