Ai broadcasts large language model weights to edge devices for on the fly processing
AIR-LLM: Broadcasting AI Weights over Radio for Memory-Free Edge LLM Inference via RF Computing
Information TheoryEmerging TechnologiesMachine Learning
Summary
Large language models are growing so big that many small devices cannot store or run them efficiently. The authors show a way to send these model parts over radio waves to devices without needing to store them in memory. These devices process the data as it arrives, saving energy and time. This method works for many users at once and reduces the time and power needed for language understanding tasks.
What this means in practice
- •For mobile network operators: Provide large language model inference services on edge devices by broadcasting model weights through existing 5G infrastructure without user-specific reconfiguration.$Commercial implications: Enables selling edge AI inference services to users without requiring local model storage, reducing device requirements and improving scalability.
- •For iot device manufacturers: Design memory-constrained IoT devices to perform advanced language tasks by processing incoming AI model weights directly through RF signals.
Authors
Zhihui Gao, Tingjun Chen, Dirk Englund
Abstract
Next-generation large language models (LLMs) are expanding from the cloud to ubiquitous edge devices. However, edge devices typically either lack the memory to store increasingly large LLM weights or, even with enough memory, spend unaffordable energy on loading the weights. This raises our question: can an edge device run an LLM without storing or loading its weights, but receive them over the air and consume them on the fly? Inspired by wireless broadcasting, we present AIR-LLM, an LLM inference architecture for edge devices, which is composed of: (i) a central radio (e.g., 5G base stations) that broadcasts the LLM weights into the air, and (ii) the edge user that receives the weights and completes the general matrix-vector multiplication (GEMV) of LLM inference directly in the radio frequency (RF) domain using RF mixers. To further shorten the airtime, AIR-LLM exploits MIMO spatial multiplexing and proposes an energy-efficient precoder-postcoder pair on the edge to calibrate its own wireless channel. Since the central radio stays user-unaware, AIR-LLM is user-scalable so that one broadcast serves unlimited users within its coverage. We implement AIR-LLM on the NVIDIA Sionna ray-traced channels of two real-world urban scenes and the profiling of a real RF mixer. With a WikiText-2 perplexity degradation of 4.0% on LLaMA-3.1-8B, AIR-LLM saves the energy by 157.7x/40.4x against the FP16 and weight-only quantization baselines; with 20 users, its airtime is 104.1x/26.0x shorter, respectively.