Swarm telepathy boosts multi-agent AI communication speed and accuracy
Swarm Telepathy: Prefill-Free KV-Cache Transfer Across Families
Machine Learning
Summary
Large language models (LLMs) working together usually communicate by sending text summaries, which is slow and wastes computing power. The authors introduce SwaT, a new way to share internal memory between different types of LLMs without needing to send text first. This method works faster and more accurately than traditional text messaging between AI agents and even helps transfer image understanding to text-only models. SwaT only needs limited training and keeps the original models unchanged.
What this means in practice
- •For ai system architects: Speed up and improve accuracy of data exchange in multi-agent AI systems using cross-family KV-cache transfer.
- •For ml engineering teams: Enable enhanced communication between vision-capable and text-only AI models without additional fine-tuning or text prefills.
Authors
Abdulrahman Alabdulkareem, Irene Tenison, Yang Zhang, Yada Zhu, Boris Katz, Dan Gutfreund
Abstract
LLM multi-agent systems split work across agents that communicate through text. An agent summarizes its findings, another agent receives the text. This constrains communication to discrete tokens and costs redundant compute, limiting bandwidth and capabilities of multi-agent systems. Recent works demonstrate KV-cache transfer between LLMs but they either require the LLMs to be within the same family or cannot function across context, thus requiring the receiver to prefill text. We present SwaT (Swarm Telepathy), the first cross-family KV-cache transfer method that allows prefill-free communication in multi-family multi-agent systems. In an agent-to-agent communication setting, combining SwaT with recent advances in KV-cache compaction beats text-to-text communication in both accuracy and speed, with a mean of $40\%$ higher accuracy and a mean of $4\times$ faster wall-clock time. Experiments show SwaT achieves high performance on 17 benchmarks across 12 LLMs and 6 families. Additionally, given a sender Vision-Language Model (VLM), SwaT is able to transfer the vision capabilities to a text-only receiver LLM exceeding text-to-text communication despite SwaT being trained on text-only data. SwaT keeps both sender and receiver models frozen without fine-tuning and only requires $<50M$ tokens and $<1$ GPU-day to train.