LLM-based Source Code Compression via Thresholded Symbol Ranking
2026-07-27 • Information Theory
Information TheoryComputation and LanguageMachine Learning
AI summaryⓘ
The authors studied how to compress computer source code without losing any information, aiming to save space in large software archives. They used Large Language Models (LLMs) to predict next symbols more efficiently by limiting predictions to the top few options and handling less likely symbols separately. Their method improved compression size and speed compared to previous LLM-based and traditional tools like zstd and bzip2. They also found that source code benefits more from their approach than regular natural language, likely because code has patterns LLMs understand better. Finally, they discuss future research directions to improve this area further.
lossless compressionsource codeLarge Language Models (LLMs)symbol rankingtop-T rankingcompression ratiocompression throughputgeneral-purpose compressorsSoftware Heritage
Authors
Angelo Nardone, Paolo Ferragina
Abstract
We study the problem of lossless compression of source code, motivated by the storage demands of large-scale software archives, such as Software Heritage (https://www.softwareheritage.org/). General-purpose compressors (e.g., zstd, bzip2) offer a good trade-off between compression ratio and speed, but fail to exploit all special regularities inherent in source code. Recent approaches leverage Large Language Models (LLMs) within Shannon's symbol-ranking framework, relying on a scheme in which the predicted rank can grow arbitrarily. While effective at reducing space, this setting incurs significant throughput degradation, and leaves open the question whether it is necessary to explicitly encode all ranks. In this work, we introduce LLM-based compressors deploying two novel symbol-ranking variants that bound predictions to the top-$T$ ranks ($T=1$ or $63$), with out-of-threshold symbols stored as exceptions and compressed jointly with the rank stream via general-purpose compressors. We conduct the first large-scale evaluation of LLM-based source code compression across 30 LLMs, including general-domain, code-specialized, and quantized models. Our $T$-bounded approach outperforms prior LLM-based compressors both in compression ratio (up to 37% relative improvement) and compression throughput (40% faster). Compared to general-purpose compressors (e.g., zstd, bzip2), we obtain up to 82% relative compression gain but at a lower speed, thus offering a new trade-off point in the compression-speed spectrum. We also show that these gains are stronger on source code than on natural language, suggesting an interesting indication, namely that source code exposes regularities captured by LLMs but missed by general-purpose exact-match-based compressors. We conclude by commenting on open problems that offer theoretical and practical avenues of research.