Lossless compression method reduces GPU memory for AI training
LCT: Losslessly Compressed Tensors with GPU friendly execution for Training
Machine Learning
Summary
Training AI models uses a lot of computer memory, partly because data needs to be stored and repeatedly compressed and decompressed. The authors created a new method called Losslessly Compressed Tensors (LCT) that stores this data in a smaller space without losing any information. They designed it to work fast on GPUs by using efficient coding techniques. Their method cuts memory use by about a quarter but slows training steps slightly. This allows bigger AI models or longer training sequences on the same hardware.
What this means in practice
- •For deep learning engineers: Reduce GPU memory usage during training of large neural networks by applying lossless compression that fits training data more efficiently without loss.
- •For ai model fine-tuning teams: Enable fine-tuning of large language models with longer context windows on existing GPU hardware by decreasing memory needed for optimizer and activation storage.
- •For cloud gpu providers: Offer enhanced training capabilities by integrating LCT to allow customers to run larger or longer AI training tasks on rented GPUs with lower memory demand.$Commercial implications: This method supports commercial GPU services by expanding capacity for advanced AI training jobs and differentiating product offerings.
Authors
Max Zhu, Bohang Sun, Pietro Lio, Jialin Yu
Abstract
Lossless tensor compression could reduce memory usage during training, but training requires tensors to be repeatedly re-encoded. We introduce Losslessly Compressed Tensors (LCT), a GPU execution framework designed for this repeated encode-decode regime. LCT exploits the natural redundancy in floating point data to reduce storage for weights, activations and optimizer states. Highly optimized GPU kernels implement Huffman coding with precomputed codebooks for low compression and decompression overhead, essential for training. Our analysis characterizes LCT's storage costs and adaptation to magnitude shifts. LCT achieves a compression ratio of up to $1.455\times$, and is demonstrated in training and finetuning settings. On a 162M-parameter NanoGPT, LCT reduces memory usage by $24.6\%$ for a $7.2\%$ step-time increase. In Qwen3-4B fine-tuning, LCT saves up to $23.8\%$ of peak memory, enabling a $3.5\times$ larger context length in training on the same GPU.