Ternary large language models store weights more efficiently with new layout
Breaking the 1.58-bit Barrier for Ternary LLMs
Artificial IntelligenceMachine Learning
Summary
Ternary large language models store numbers as one of three symbols: -1, 0, or +1, typically requiring about 1.58 bits per number. The authors found that many models have lots of zeros and introduced a new method called BITCOS that stores these numbers more efficiently by separating presence and sign information. This method uses fewer bits per weight and speeds up processing on modern CPUs and GPUs. Tests showed improved speed and reduced storage compared to common methods across different hardware platforms.
What this means in practice
- •For ai system engineers: Implement more storage-efficient and faster ternary weight formats for large language model inference on CPUs and GPUs.
- •For embedded device developers: Use reduced-bit-weight storage methods to run large language models in resource-constrained environments with faster processing and smaller memory footprints.
Authors
Evangelos Georganas, Alexander Heinecke, Pradeep Dubey
Abstract
Ternary Large Language Models (LLM) store every weight as one of three symbols $\{-1,0,+1\}$, so the cost of a ternary model is conventionally referenced to the information-theoretic $\log_2 3 \approx 1.585$ bits per weight. The prevailing deployment format packs five ternary weights into one byte (five-trit packing), and due to the power-of-two group sizes used in practice this rounds up to $1.625$ bits per weight. This effective storage bit-width treats the three symbols $\{-1,0,+1\}$ as equiprobable. We measure the actual symbol distribution of 29 ternary LLM models and find that zeros account for up to $51.5\%$ of all weights. Motivated by this finding, we introduce BITCOS, a simple distribution-adaptive layout comprised of a dense presence bitmap plus a compacted sign vector, and costs $2 - z$ bits per weight element given a zero density $z$ in the model's weights. BITCOS stores weights more compactly than the five-trit packing in 26 of the 29 tested models, and reaches $1.485$ bits per weight on the sparsest of them. BITCOS is amenable to efficient unpacking on modern processors and GPUs, and we present optimized unpacking sequences for AVX-512, AVX2 and Intel Xe2 GPUs. Measured against production state-of-the-art ternary matrix-vector multiplication kernels, at the zero densities real-world ternary models exhibit, the realized gain with our proposed layout is up to $1.28\times$. Finally, we illustrate end-to-end LLM inference results on 5 different platforms (client and server CPUs, integrated and discrete Xe2 GPUs) where decode throughput improves by up to $1.18\times$ on CPUs and $1.27\times$ on GPUs.