Kafila accelerates large language models on trusted personal device networks
Kafila: Serving Large Language Models on a Trusted Set of Heterogeneous Commodity Machines
Distributed, Parallel, and Cluster ComputingMachine Learning
Summary
Running large language models usually requires big computers, but many people only have smaller personal devices. The authors created Kafila, a system that combines these trusted devices, even if they are different and behind firewalls, to work together efficiently. Kafila carefully splits the model based on each device's speed and memory so that no device slows down the whole process. This makes running big AI models on a trusted group of smaller computers much faster and more practical than before.
What this means in practice
- •For small research groups: Run large language models collectively on personal devices while ensuring all are trusted and optimally utilized for faster results.
- •For distributed computing engineers: Use Kafila’s model division and network setup to build efficient multi-device AI inference pipelines across mixed hardware behind NATs.
Authors
Murtaza Rangwala, Richard O. Sinnott, Rajkumar Buyya
Abstract
Between them, the members of a research group or a circle of friends own several consumer computers, none large enough to run a capable large language model. Existing systems pool such capacity across open swarms anyone may join, which a group admitting only trusted machines cannot use. Bounding membership removes what they depend on: a swarm holds each part of the model on several peers and routes around a slow one. A bounded session must use every device it admits. Its pipeline advances at the pace of whichever device received a share it cannot serve quickly, so the division has to be right before serving begins. We propose Kafila, whose protocol assembles a ring from behind NATs, preferring direct paths and relaying where traversal fails, while its planner measures each device's memory bandwidth, capacity and reachability, divides the model exactly for a fixed ring order, and places the head, which holds the embedding and output projection, together with that division rather than beforehand. On machines with different capabilities across three fleets, from a shared LAN to five devices spanning two continents, Kafila shortens the slowest pipeline stage by up to $5.2\times$ against the even split of pipeline parallelism, as in GPipe, and up to $3\times$ against the memory-proportional split of personal-device inference, as in exo, keeps 75 to 87 per cent of the committed hardware doing work where those divisions fall below half, and serves a model no uniform split can place on the fleet at all. What that is worth to a user depends on how much of a token is computation rather than network. Where the members share a network the same division returns $1.56\times$ the throughput of a uniform split and $1.25\times$ of a memory-proportional one, and under four concurrent users that lead compounds to $3.2\times$ rather than fading, each user served at almost the rate of one.