ThinkBit cuts AI model size while keeping reasoning strong
ThinkBit: Search Codewords, Recalibrate Scales, Select by Reasoning
Artificial IntelligenceComputation and Language
Summary
Big AI language models that can think carefully usually take up a lot of computer memory. This paper presents ThinkBit, a new way to shrink the memory needed by storing parts of the model using just two bits instead of larger formats. The authors show that ThinkBit keeps the model’s ability to solve tough math problems nearly as well as the full-size version, even though the model is much smaller. They do this by smartly picking how to store parts of the model, adjusting internal scales, and fine-tuning with special training.
What this means in practice
- •For ai infrastructure engineers: Reduce memory and bandwidth costs of running large language models without major drops in complex reasoning accuracy using two-bit weight storage.
- •For mobile app developers: Deploy smaller, more efficient AI models with preserved calculation abilities for devices with limited memory and bandwidth.
Authors
Rhea Sanjay Sukthanker, Pashmina Cameron
Abstract
Reasoning language models generate thousands of tokens per answer, and generation speed is limited by memory bandwidth, so storing weights in two bits instead of bfloat16 can potentially make inference 8$\times$ cheaper. Two-bit methods are usually judged by perplexity and multiple-choice question-answering accuracy, both of which say little about whether a model can still reason through more difficult problems. We present ThinkBit, a strictly scalar two-bit framework built to preserve reasoning, in which every weight is an integer codeword times a bfloat16 scale shared by a group of weights. We first use a sensitivity metric to assign each layer a ternary, 2-bit, or 3-bit tier to allocate bitwidths optimally among layers. This is followed by a layerwise Gumbel-softmax-based scalar quantization algorithm that searches for optimal codewords block by block. Since a layerwise scheme does not capture how quantized layers interact with each other, we recalibrate scales, with the codewords frozen, using a knowledge-distillation-based loss against the teacher model. We then perform a short quantization-aware training (QAT) stage that lets the model mimic the base model's distribution by reassigning codewords. Finally, a paired reasoning test on training problems selects the final checkpoint between the recalibration stage and the QAT stage. On Qwen3-1.7B at matched size, our recalibrated checkpoint scores 46.7 on MATH-500 against 30.0 for EfficientQAT trained on the same data, and 73.0 after QAT on reasoning traces; at 0.52 GB it is smaller than every published 2-bit GGUF of the model yet more accurate on MATH-500. We show that this framework scales and up to 14B, a 3.74 GB model scores 94.8 on MATH-500, against 97.5 for the 29.5 GB bfloat16 model.