BitNet inference speeds up language model decoding with signed int4 instructions
Implementation and Evaluation of BitNet Inference on a CGLA by Signed-Int4 Instructions
Hardware Architecture
Summary
Running large language models is slow and uses a lot of energy because it moves lots of data for each word it predicts. The authors show how to speed up a specific type of model called BitNet by using special small 4-bit integer math instructions on a custom chip design. This mapping allows the model’s math to run efficiently without needing a special custom processor. Their tests show the chip can handle over two words per second with this approach.
What this means in practice
- •For chip designers: Implement low-bit arithmetic for language model inference using signed int4 instructions on programmable ASICs for energy-efficient decoding.
- •For embedded system developers: Offload BitNet model inference to hardware with local memories and integer lanes to boost token processing speed on embedded platforms.
Authors
Takuto Ando, Yasuhiko Nakashima
Abstract
Large language model (LLM) inference transfers model weights and activations for every generated token, making memory traffic and its energy cost part of the decode path. BitNet b1.58 represents its low-bit weights by ternary values and uses integer activations. However, this arithmetic does not match conventional int8 or floating-point general matrix multiplication, and existing BitNet accelerators implement it in specialized datapaths. We instead map this operation to a CPU-Grounded Linear Array (CGLA), a programmable ASIC with explicit direct memory access, local memories, and reusable compiler-visible integer lanes. The mapping adds OP_SMA4 as a reusable signed-int4 multiply-accumulate instruction rather than a BitNet-only datapath. Each ternary weight occupies one signed 4-bit lane. Each int8 activation is split into two signed-int4 fragments and reconstructed by shift-and-add. Frequency scaling of the 145 MHz FPGA measurement to an 840 MHz 28 nm CGLA achieved 0.390 ns per signed-int4 product. We showed that CGLA-offloaded BitNet C++ execution measures 2.52 tokens/s.