Figflex accelerator boosts mixed precision large language model matrix math
FIGFLEX: A Mixed-Precision FP-INT GEMM Accelerator Co-Designed with Low-Bit LLM Quantization
Hardware ArchitectureMachine Learning
Summary
Running large language models (LLMs) usually involves multiplying big tables of numbers, which uses a lot of memory and computing power. The authors show a new hardware design called FIGFLEX that works faster when using fewer bits to store numbers, especially integer weights, without losing accuracy. They also created a matching method to squeeze numbers into smaller sizes that fit perfectly with the hardware, keeping the model's performance at least as good as existing GPUs. On tests with a popular LLM, FIGFLEX improved processing speed while keeping or improving prediction quality compared to prior methods.
What this means in practice
- •For chip architects: Design energy-efficient accelerators that speed up mixed-precision matrix multiplications in large language model inference hardware.
- •For ai infrastructure teams: Deploy faster and more memory-efficient hardware to run quantized large language models at lower cost and power without losing accuracy.
Authors
Faraz Tahmasebi, Haocheng Xu, Andrew Trauger, Yomal Perera, Zhenyu Tang, Nael Chowdhury, Sitao Huang
Abstract
Weight-only quantization keeps LLM activations in floating point and stores weights as low-bit integers, turning the dominant workload into floating-point-integer (FP-INT) general matrix multiplication. Recent quantizers reach a 2-3-bit average with mixed precision, but hardware has not kept pace: GPUs dequantize weights to FP16 before the tensor-core multiply-accumulate, and bit-parallel FP-INT accelerators run at a single weight precision, so fewer bits save memory but not computation. In this paper, we present FIGFLEX, an algorithm-hardware co-designed mixed-precision accelerator whose bit-parallel datapath scales throughput as precision falls. FIGFLEX contributes (1) a shift-add, radix-4 reformulation of FP-INT multiplication that fuses the multiply-accumulate into a unified, precision-flexible partial-product reduction and roughly halves the partial products, together with a novel INT3 optimization that reclaims the idle datapath radix-4 leaves at that precision; and (2) a mixed-precision quantizer that assigns each channel group its bit-width according to the error GPTQ cannot compensate, with a per-input-channel column scaling whose power-of-two normalization keeps the scaled activations out of the FP16 subnormal range, so that the quantized model maps directly onto FIGFLEX's datapath with the same accuracy as on a GPU. On LLaMA-2-7B, FIGFLEX attains 5.85 and 8.77 WikiText-2 perplexity at 3- and 2-bit average precision, lower than GPTQ, AWQ, SliM-LLM, AMQ, and QuIP# (without fine-tuning), while delivering 1.56x and 1.42x higher whole-chip throughput per area on the linear GEMMs of a 3-bit mixed-precision model than AxCore and FIGLUT, respectively.