Continuous benchmark measures autonomous site reliability agents performance
SRE-Marathon: A Continuous, Change-Driven Benchmark for Autonomous Site Reliability Agents
Artificial IntelligenceSoftware Engineering
Summary
Managing computer systems involves fixing problems that can happen anytime and overlap each other, which is hard to test with traditional one-at-a-time challenges. The authors created SRE-Marathon, a continuous test that mimics real system issues by injecting overlapping faults and changes over time. It scores agents on how well they notice, locate, and fix these problems while everything runs. Their tests show even the best agents struggle to complete repairs when issues overlap and persist.
What this means in practice
- •For site reliability engineering teams: Evaluate and improve automated responders by testing their ability to handle multiple overlapping faults in realistic cloud deployments.
- •For cloud infrastructure operators: Use continuous simulated fault scenarios to validate and stress-test incident management tools during live system operation.
Authors
Yifang Tian, Yingjian Bai, Yifeng He, Zichun Chong, Yuanchen Gao, Yiran Li, Hans-Arno Jacobsen
Abstract
Benchmarks for site reliability engineering (SRE) agents are typically episodic: one fault is injected, the agent receives an incident task, and its response is scored. Production operation is not. Incidents surface through noisy alerts, overlap in time, and often originate from code or configuration changes. We present SRE-Marathon, a benchmark for long-horizon, continuous SRE operation. An agent is invoked at a fixed cadence with cumulative alert history and a persistent workspace while operating a live two-zone Kubernetes deployment as a fault orchestrator injects overlapping faults according to a seeded, production-calibrated schedule. Curated code and configuration changes deployed through the same build pipeline available for repair. Each run is recorded into a sealed bundle and scored offline: Marathon-Score credits each injected fault for ordered progress through correlation, localization, and repair, with all metrics computed deterministically from recorded system evidence. Across three applications and about sixty faults per run, the best of 10 methods reaches only 41.3 out of 100. Agents often correlate and localize faults, but almost never complete repairs while the faults remain active.