Benchmark measures collaboration in asynchronous multi-agent coding tasks
AsynCodeBench: Benchmarking Collaboration of Asynchronous Multi-Agent Systems in Software Engineering
Software EngineeringArtificial IntelligenceMultiagent Systems
Summary
When many computer programs work together to write code, it can be tricky to tell how well they actually coordinate with each other versus just how good they are individually. The authors created a new way to measure teamwork by breaking coding tasks down into smaller parts that depend on each other. They track when and how these parts get completed to see how well different agents work together. Their experiments show that just being good at coding alone does not mean the agents collaborate well, and teamwork often happens in sudden bursts rather than gradually.
What this means in practice
- •For software engineering teams: Evaluate and improve coordination in teams of AI coding assistants working asynchronously on complex software projects.
- •For software testing teams: Use dependency-based metrics to pinpoint coordination bottlenecks when multiple tools contribute to a software build or testing process.
Authors
Kaituo Zhang, Zhen Xiong, Zhimeng Jiang, Mingyu Zhong, Zhouyuan Yuan, Zhecheng Li, Bowen Lin, Chia-Yuan Chang, Mingzhi Hu, Huazheng Wang, Ying Lin
Abstract
Multi-agent coding has emerged as an increasingly active direction in software engineering, where complex development tasks are decomposed across multiple specialized agents working on different parts of the problem. Despite the shift from individual problem solving to distributed collaboration, multi-agent systems still lack a direct measure of collaboration and are largely evaluated through task-level outcomes inherited from single-agent coding, conflating individual coding capability with cross-agent coordination. We introduce AsynCodeBench, a dependency-centric benchmark for asynchronous multi-agent software engineering that represents each task with an explicit dependency graph and executable Dependency Checkers. Through this dependency-tracking process, we propose two complementary measures: Asynchronous Dependency Pass Rate (ADPR), which measures how many cross-agent dependencies are ultimately satisfied, and Dependency Resolution Step (DRS), which measures when each dependency first becomes satisfied during execution. AsynCodeBench comprises 19 tasks from real-world repositories, exposing 52 directed dependencies as explicit units for evaluating cross-agent collaboration. Experiments across model families, scales, and generations reveal a clear gap between coding and collaboration capability: improvements in coding performance do not necessarily translate into stronger collaboration, and task-level metrics can diverge substantially from dependency-level collaboration measures. Dependency-trajectory analysis further reveals that successful coordination often emerges not gradually, but through concentrated bursts in which many dependencies become resolved over a short portion of the execution trajectory, a pattern we term a hopping window.