Text compression method matches lossless quality with fewer bits
Lossy Compressive Text Autoencoders
Computation and Language
Summary
Compressing text usually means making it smaller without losing any information. This paper shows a way to shrink text into fewer bits by learning a compact internal representation that can be turned back into text pretty well. The authors compare different techniques for this compression and test how close the recovered text is to the original, both in word choice and meaning. Their method achieves similar file sizes to traditional lossless compressors but still works well when the text is used for tasks like answering questions or comparing meanings.
What this means in practice
- •For natural language processing teams: Create smaller text embeddings to speed up storage and transfer while maintaining accuracy for downstream language tasks.
- •For web service operators: Reduce bandwidth and storage costs by employing compact text encoding that preserves semantic content in transmitted data.
Authors
Vinko Sabolčec, Angelos Katharopoulos, David Grangier
Abstract
Our work explores learning a compressed latent representation of text, at the intersection of data compression and representation learning. We propose an autoencoder architecture that performs residual downscaling and upscaling of hidden representations along the time axis, with a residual low-dimension discrete bottleneck. We analyze our approach for different quantization methods, training objectives, and datasets. For different levels of compression, we evaluate the similarity between the original and reconstructed text both at the surface-level (BLEU) and at the semantic-level (LLM-based judge). Additionally, we evaluate our models on downstream question-answering and semantic text similarity benchmarks. Our approach results in compressed representations which are on par with lossless text compression algorithms at 2.24 bits per byte on web text data, while having good reconstruction and downstream task performance.