BitMTP accelerates low-bit large language models for faster text generation
BitMTP: When Multi-Token Prediction Meets Low-Bit Large Language Models
Computation and LanguageArtificial IntelligenceDistributed, Parallel, and Cluster ComputingMachine Learning
Summary
Large language models can be slow and costly to run because they process text one token at a time and use big numbers to store data. The authors found a way to speed this up by combining a type of model that predicts several words at once with models that use smaller numbers to save space. Their method, BitMTP, keeps the text prediction accurate while running much faster on different types of GPUs. This means applications using language models can respond quicker without needing more powerful hardware.
What this means in practice
- •For machine learning engineers: Deploy smaller, faster language models with multi-token prediction to reduce latency in AI-powered software.
- •For edge device developers: Run efficient low-bit language models with BitMTP on limited hardware like Jetson Orin for real-time AI applications.
Authors
Ning Zhang, Shihao Wang, Jinrui Zhang, Chaodong Xiao, Lei Zhang
Abstract
For efficient deployment of large language models (LLMs), methods of ultra-low-bit backbones reduce weight storage, memory traffic, and linear-layer execution cost, while multi-token prediction (MTP) methods reduce sequential decoding steps by predicting multiple future tokens in parallel. Although complementary, the direct combination of these methods cannot accelerate decoding at comparable quality: masked drafting lacks intermediate token input supplied in next-token training, while wider computation and causal key-value (KV) updates can offset the savings from fewer steps. To address these challenges, we propose BitMTP, an algorithm-system co-design framework for efficient MTP on ultra-low-bit LLM backbones. A joint next-token prediction and MTP objective equips a single model with both autoregressive (AR) generation and block-causal drafting. Confidence-only decoding, speculative-only decoding, and hybrid decoding provide different quality-throughput operating points without using separate draft and verifier models. Accepted-prefix alignment keeps the committed prefix, materialized causal KV boundary, and CUDA Graph replay state consistent for continued generation. BitMTP uses this contract to select width-specialized low-bit execution and reuse captured graphs over shared KV storage. On NVIDIA H20, BitMTP with hybrid decoding achieves five-task mean speedups of 2.2x on BitNet b1.58-2B and 2.1-2.4x on 1.7B-4B Bonsai models at comparable task quality. On RTX 3090 and Jetson Orin, it further achieves up to 2.0x in five-task mean throughput and 3.3x on MBPP+, respectively. These results demonstrate that BitMTP accelerates ultra-low-bit LLM generation across backbones, model sizes, and GPU types. The source code is released at https://github.com/PolyU-VCLab/BitMTP