Bit-sharing boosts large language model decoding speed with new hardware
BS-Spec: BFP-Based Bit-Sharing for Self-Speculative LLM Decoding with LUT-Driven Mixed-Precision Acceleration
Hardware Architecture
Summary
Running large language models (LLMs) can be slow because they handle complex decoding steps, especially for varied request sizes and demands. The authors propose BS-Spec, a method that shares parts of model data in fewer bits and smartly picks precision levels depending on the workload, all while using a special hardware design. This lets the system do most of the work faster without needing extra copies of the model. Their tests show this approach speeds up common decoding methods by nearly twice and improves on similar newer methods by over 60%.
What this means in practice
- •For ai service providers: Improve throughput of language model inference for varied workloads by dynamically adjusting precision and sharing bits without storing multiple models.
- •For hardware designers: Use the proposed mixed-precision BFP accelerator design with lookup-table control to optimize hardware for self-speculative LLM decoding.
Authors
Jiwoo Kim, Gunho Park, Sunwoo Yoo, Baeseong Park, Dongsoo Lee, Youngjoo Lee
Abstract
As large language models (LLMs) are increasingly deployed across diverse inference services, serving workloads span a wide range of batch sizes, context lengths, and performance requirements. Speculative decoding improves inference throughput, but conventional approaches require a separate draft model and KV cache, while existing self-speculative methods suffer from workload-insensitive draft configurations and inefficient mixed-precision execution. In this paper, we present BS-Spec, a block floating point (BFP)-based bit-sharing framework that jointly optimizes draft construction, workload-aware precision selection, and hardware execution. BS-Spec derives multiple draft representations directly from a single BFP target model by partially loading the most significant bits of weights and the KV cache while preserving shared exponents, eliminating separate draft-model storage. Offline calibration selects the weight and KV-cache precision pair that maximizes estimated speedup for each serving regime, and a precision lookup table applies the selected configuration at runtime. To support this adaptive execution, we design a mixed-precision BFP accelerator with a lookup table (LUT)-based variable-precision datapath for both draft and target computation. Experimental results show that BS-Spec achieves up to $1.9\times$ speedup over conventional autoregressive decoding and improves speedup over the evaluated bit-sharing self-speculative decoding baselines by $67.2\%$ on average, demonstrating the benefits of jointly co-designing bit sharing, precision adaptation, and specialized hardware for diverse LLM serving workloads.