Sparta boosts long-context LLM speed and energy efficiency on edge devices
SPARTA: A Sparse Ternary LLM Accelerator with Adaptive Attention and Unified Datapath for Long-Context Edge Inference
Hardware ArchitectureArtificial Intelligence
Summary
Large language models can be too big and slow to run on small devices because of memory and processing limits. The authors created Sparta, a special chip accelerator that uses clever tricks to shrink the data and reduce unnecessary work during attention calculations. This lets Sparta handle long text inputs faster, using less energy, all on devices with limited resources. It also combines different tasks into one unified system to avoid extra hardware costs and delays.
What this means in practice
- •For edge device engineers: Integrate Sparta accelerator designs to run long-context language models efficiently on resource-limited edge hardware.
- •For embedded system developers: Build smarter assistants and inference tools that handle longer text inputs without increasing hardware size or power use on embedded platforms.
Authors
Yanzhen Zhou, Zhengyu Han, Qunyou Liu, Zheyu Yan, Weihong Xu, Cheng Zhuo
Abstract
LLMs on edge devices are constrained by limited memory and bandwidth. Although 1.58-bit ternary weights greatly reduce model storage, long-context inference is increasingly bottlenecked by KV-cache traffic and attention. Existing accelerators also use separate datapaths for ternary linear layers and attention, incurring hardware and execution overhead. We present SPARTA, a sparse ternary LLM accelerator for efficient long-context edge inference. SPARTA integrates: (1) a phase-aware sparse dataflow exploiting heterogeneous sparsity across compute-bound prefill and memory-bound decode; (2) a unified compute fabric mapping ternary SpMM/SpMV and bit-serial attention onto shared processing elements to exploit zero-weight and zero-bit sparsity; and (3) an adaptive two-stage attention engine combining page-level Top-k selection to reduce K-cache accesses with token-level Top-p pruning to reduce V-cache traffic. SPARTA reduces KV traffic by 4.1-4.8x over full attention and 1.3-1.5x over a state-of-the-art sparse-attention scheme, with task-dependent quality loss. Compared with NVIDIA Jetson AGX Orin, SPARTA achieves 3.9x higher generation throughput and 8.3x higher end-to-end energy efficiency on average; versus AMD Ryzen 7 9700X, the gains are 5.0x and 81.7x. It also delivers 1.6-12.0x higher throughput, 2.0-4.2x higher energy efficiency, and 1.3-51.4x higher area efficiency than FIGLUT, TeLLMe, and MCBP.