Fine-Tuning Low-Bit Models with Gradient in Quantized Code Space
2026-08-31 • Machine Learning
Machine Learning
AI summaryⓘ
The authors focus on fine-tuning models that use very low-bit (compressed) numbers to save memory and speed up use. They point out that existing methods either make the training less accurate or take too long. To fix this, they introduce a new way called 'code surrogate gradient' that speeds up learning while keeping the model faithful to how it will run when deployed. Their method, GradCodes, works well on several language tasks and different types of low-bit models. They also provide the code for others to use.
low-bit quantizationfine-tuningquantization codessurrogate gradientdeployment faithfulnessoptimizationarithmetic reasoninginstruction followingstructured language understandingGradCodes
Authors
Shiguang Wu, Zhouchen Lin, Quanming Yao
Abstract
Fine-tuning Low-bit models aims to adapt a quantized model while keeping the final deployed checkpoint in the same low-bit form. This setting is practically important as it reduces memory and inference cost for storage and deployment. Under this constraint, adaptation becomes an optimization problem over quantization codes and scales. Existing continuous low-bit training is efficient, but it can be distorted by straight through estimation error or by post-quantize gap; discrete search is deployment-faithful, but it is often too inefficient under a finite training budget. We propose code surrogate gradient as the first order signal in deployable code space to acceleate optimization, and performing guided search to preserve deployment faithfulness. Experiments across arithmetic reasoning, instruction following, and structured language understanding show that GradCodes consistently improves fine-tuning low-bit models across different quantization datatypes. Code is provided at https://github.com/ovo67/GradCodes.