Summary
Software can break over time and stop working, especially when important parts go missing or are outdated. This paper introduces ReviveBench, a set of challenges that test if coding agents can fix broken software and rebuild complete software engines from scratch. The researchers found that some advanced coding agents can pass most of these challenges, even when the original code looks very different or when parts are purposely hidden. They also discovered that the tools used to check if the software works correctly sometimes miss errors, so they suggest practical ways to improve testing. Overall, this work shows how coding agents might help restore and recreate software in the future.
What this means in practice
- •For software maintenance teams: Automatically restore legacy software with broken dependencies or missing code modules to working condition using coding agents evaluated by ReviveBench.
- •For industrial software developers: Rebuild complex industrial software engines from open descriptions, improving productivity in reconstructing systems like CAD or CRM platforms.
Authors
Tianyu Liu, Dingyuan Dai, Yufan Du, Zhen Yang
Abstract
Can coding agents restore software that no longer runs while preserving its underlying methods, and reconstruct industrial software engines from open specifications? Here we introduce ReviveBench, a benchmark with two task families evaluated by hidden verifiers calibrated against native execution environments, established engineering tools, or purpose-built reference implementations. The revival family comprises ten tasks involving dependency incompatibilities, deleted core modules, legacy builds, and a GPU-based foundation model. Every starting workspace fails verification, and the strongest evaluated model passes all ten tasks in at least one run each. In contamination-control experiments, identifier obfuscation reduces line similarity to the original implementations from 0.51--0.96 to 0.03--0.44 without reducing the observed pass rate of any evaluated model. On repositories created after the stated knowledge cutoffs, the strongest model passes eight of nine runs. The reconstruction family comprises thirteen tasks spanning numerical, geometric, hardware, and transactional systems (e.g. CAD and CRM). Two models meet the benchmark's pass criteria on all thirteen, although our audit shows that the CFD task cannot establish numerical-solver capability. Benchmark construction and auditing uncover 28 verifier defects, including 24 false negatives and two false positives. These findings show that executable verification can itself introduce substantial measurement error. We present three practical checks: test whether prescribed methods can reach the grading thresholds, investigate agreement among independently generated candidates, and recompute diagnostics from submitted artifacts. ReviveBench thus provides both an evaluation of software revival and engine reconstruction, and cases in validating the verifiers used to measure coding agents for software design.