Language model token compression technique reduces quality loss
zip2zip++: Towards Inference-Time Token Compression with Minimal Quality Loss
Computation and Language
Summary
Compressing text tokens can make language models run faster but often lowers their quality. The authors found that this loss in accuracy isn’t unavoidable and improved the process by fixing key issues in the compression steps, without changing the original model. They made the system smarter about how tokens are combined and represented, which helped keep the model’s understanding sharper even when using fewer tokens. Their approach improved performance across different models and tasks, getting closer to uncompressed quality at similar token reduction.
What this means in practice
- •For ai engineers: Enable faster language model inference by reducing token count without major loss in reasoning or accuracy.
- •For cloud service providers: Lower compute costs when serving text-based AI models by integrating adaptive token compression that maintains quality.
Authors
Saibo Geng, Andrea Gentilini, Xinxian Ma, Nathan Ranchin, Yunzhen Yao, Robert West
Abstract
Dynamic vocabularies built on-the-fly via Lempel-Ziv-Welch (LZW) compression let a language model represent the same text in fewer tokens, but prior work shows that this efficiency comes at a cost to reasoning and perplexity. We show that this quality gap is not a fundamental price of compression but can be substantially closed through targeted improvements to the compression layer and LZW codec, without modifying the underlying pretrained language model. Our analysis reveals four key issues: (1) LZW's compression algorithm creates an implicit conflict between the input embedding and the output embedding, a tension that a single shared embedding cannot resolve; (2) naively applying RoPE to LZW-compressed sequences ignores that merged tokens can span multiple tokens; (3) merging certain tokens (notably digits) disproportionately hurts downstream accuracy; and (4) dynamic vocabularies introduce an inherent segmentation ambiguity that confounds standard evaluation. We address these through decoupled input/output dynamic embedding layer, size-aware RoPE positioning, and a modified LZW codec supporting selective merges and multi-view segmentations. Across four models, our recipe improves perplexity on every corpus we evaluated as well as mathematical reasoning accuracy, closing roughly half the quality gap between zip2zip and the uncompressed baseline at a similar compression ratio on general text. This represents a significant step toward lossless adaptive-vocabulary compression for language models.