Benchmark measures how multi-agent tasks share and isolate information
CoMemBench: Benchmarking Collaborative Memory Boundaries across Multi-Agent Workflow Topologies
Artificial Intelligence
Summary
When multiple agents work together on a task, they need to share the right information but keep unrelated details separate. The authors call this controlled sharing a collaborative memory boundary. They created CoMemBench, a big test set that checks how well workflows share and isolate information depending on their connection shape. Their tests show that sharing more information helps agents get the job done but can also let unwanted or incorrect info slip through.
What this means in practice
- •For workflow engineers: Evaluate how task designs affect information flow and isolation between agents in multi-step processes.
- •For distributed system developers: Test reliability and isolation robustness of memory sharing mechanisms across different multi-agent topologies.
Authors
Sen Zhao, Ruiqi Kong, Zuyu Zhang, Lifeng Shen, Xinyu He, Ding Zou, Xu Zhang, Qinghua Zhang
Abstract
Multi-agent workflows require task-relevant information to be shared across agents, while irrelevant, stale, unverified, or incompatible information must remain isolated. We call this task-conditioned scope of information a collaborative memory boundary. Workflow topology determines which intermediate artifacts are applicable to which downstream workers and when they cease to be valid, thereby providing a structural stress dimension for sharing and isolation. Existing memory benchmarks primarily evaluate retention and retrieval, whereas multi-agent benchmarks emphasize coordination and end-to-end completion, leaving topology-conditioned memory boundaries largely unmeasured. We introduce CoMemBench, an execution-grounded benchmark for collaborative memory sharing and isolation across multi-agent workflow topologies. It constructs 800 composite workflows across four domains from source-grounded dependency graphs, with node-local specifications, verifiable artifact handoffs, native evaluators, and matched isolation challenges. CoMemBench measures workflow completion, verified node progress, required-handoff reliability, isolation robustness, and token cost. Experiments reveal a sharing-isolation trade-off: broader context improves information availability but can weaken isolation, while system rankings shift across topologies and artifact violations.