BitNest speeds up memory-efficient large language model inference
BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Acceleration
Artificial Intelligence
Summary
Running large language models (LLMs) to generate text can be slow and use a lot of memory. The authors developed BitNest, a new method that speeds up text generation by guessing several words ahead using less memory. It combines low-precision and high-precision data cleverly so the model runs faster without needing extra space. Tests show BitNest works well on popular small LLMs, speeding them up while keeping the same quality.
What this means in practice
- •For mobile app developers: Accelerate on-device LLM-powered applications by reducing memory overhead and improving decoding speeds for better user experience.
- •For cloud service engineers: Optimize large language model inference on memory-limited servers by employing BitNest to gain speed without extra hardware costs.
Authors
Chence Yang, Ningxi Cheng, Arash Akbari, Qitao Tan, Qingchan Zhu, Ci Zhang, Changdi Yang, Yanzhi Wang, Wei Niu, Jinhui Wang, Jin Lu, Geng Yuan
Abstract
Speculative decoding accelerates autoregressive generation by using a lightweight draft to propose multiple tokens for parallel verification. However, existing methods often require an additional draft model or weight representation, introducing non-negligible memory overhead on resource-constrained devices. Self-speculative approaches reduce this overhead, yet still face trade-offs between draft quality, target quality, and storage efficiency. We propose BitNest, a bit-nested speculative decoding framework that embeds a low-precision draft directly into the higher-precision target representation. Instead of deriving a draft from a predefined target, BitNest first constructs a strong low-precision base and then recovers the higher-precision target through residual refinement, enabling both models to share a single physical weight representation. BitNest further extends this progressive-precision design to the KV cache for long-context inference. Across multiple 7B--8B edge-friendly LLMs and diverse workloads, BitNest achieves an average speculative acceptance rate of 95.2% while closely preserving higher-precision model quality, and delivers 1.48--1.61x end-to-end speedup over FP16 autoregressive decoding. On the LLaMA models supported by all representative self-speculative baselines, BitNest also achieves consistently competitive or higher decoding speedup.