LLM agents struggle to diagnose bugs in real distributed systems
DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems?
Software EngineeringDistributed, Parallel, and Cluster ComputingMachine Learning
Summary
Finding bugs in complex software that runs across many machines is hard. The authors created a test set called DistDebug-Bench with 120 real bugs from popular distributed systems to see how well AI agents can find these bugs. They found that current AI agents often miss problems because they don’t think step-by-step clearly and get confused by tools acting unpredictably. This shows there’s still a lot of work before AI can reliably help fix complicated system failures.
What this means in practice
- •For distributed system engineers: Evaluate and improve AI-based tools for diagnosing hard-to-find bugs across machines and services.
- •For software testing teams: Use the benchmark to test how well debugging assistants handle complex, real-world software failures.
Authors
Yuxuan Jiang, Chenglin Liu, John Rose, Mason Miller, Ruthesh Thavamani, Juntao Wu, Yinwei Huang, Jiahao Xu, Peng Huang
Abstract
LLM agents show strong potential in software engineering tasks such as competitive programming and code synthesis. Existing benchmarks, however, largely focus on small-scale settings such as self-contained Python modules, leaving distributed systems, an essential class of production software, under-examined. We introduce DistDebug-Bench, a benchmark for evaluating agents' ability to diagnose complex-system failures at the code level. It curates 120 high-quality real-world bugs from 11 widely used distributed systems, spanning 5 programming languages and 10 failure patterns. Through an extensive evaluation of state-of-the-art LLMs and agents, we identify substantial gaps between current agents and real-world debugging tasks. A recurring gap is tunnel vision due to a lack of explicit reasoning structure and nondeterminism from inconsistent tool-use scope. The benchmark and evaluator are available at https://github.com/OrderLab/distdebug-bench, with agent traces and a leaderboard at https://distdebug.com.