AI helps find faster consensus for geo-distributed systems
The End of Consensus Research?
Distributed, Parallel, and Cluster ComputingArtificial Intelligence
Summary
Consensus means getting many computers in different locations to agree on the same information, which is important but hard to do quickly. The authors show how AI helped them create a new method called A-greement that speeds up agreement in systems spread across different geographic places. This new method lowers the time it takes to do common tasks in a key-value store compared to older approaches. They also built a special AI tool named AI-greement to automatically find, test, and verify new consensus methods, uncovering some hidden problems in older ones. However, humans are still needed to guide the research process alongside the AI.
What this means in practice
- •For cloud infrastructure teams: Run geo-distributed key-value stores with lower latency by implementing the AI-discovered A-greement consensus protocol for faster data consistency.
- •For software engineers in distributed systems: Use the AI-greement framework to automatically generate, test, and verify new consensus protocols tailored to specific network and workload conditions.
Authors
Clément Burgelin, Rachid Guerraoui, Zicong Hong, Yinan Hu, Anne-Marie Kermarrec, Rafael Pires, Rasmus Moorits Veski, Martijn de Vos
Abstract
Consensus is a fundamental problem in distributed computing and at the core of state machine replication (SMR). Despite decades of active human research, discovering practical low-latency solutions to the problem is still open, as it requires combining protocol knowledge, correctness reasoning, implementation, and experimental evaluation. We report on how AI helped us discover A-greement, a new consensus protocol for geo-replicated state machines that significantly outperforms state-of-the-art solutions. Averaged across six deployments with 5 or 9 replicas, A-greement reduces the mean latency of a geo-distributed key-value store over the fastest baseline by 12.2% and 33.3% on write-heavy and read-heavy workloads, respectively, and matches the fastest baseline on read-only workloads. A-greement combines lease-authorized local reads, direct client learning for leader-ordered writes, and a fast fixed-quorum per-key write path, selected according to network topology and workload. At the heart of our discovery lies AI-greement, an agent harness for the "autonomous" discovery, implementation, experimentation, and verification of candidate protocols. AI-greement integrates (1) a concept-oriented protocol knowledge base, (2) a modular implementation skeleton, and (3) a Lean-based verification framework with reusable safety definitions, reference specifications, and proofs. AI-greement is of independent interest: its Lean formalization uncovers several previously unreported vulnerabilities in existing consensus protocols, and it provides guidelines on how domain-specific harnesses can turn general-purpose coding agents into effective tools for (distributed) computing research. Yet we argue that human consensus experts remain essential before, during, and after the search: agents reshape the researcher's role rather than end consensus research.