Dynamic semantic compression boosts large language model speed and memory
Dynamic Semantic Compression for Efficient Latent-Space Inference in Large Language Models
Computation and LanguageArtificial Intelligence
Summary
When computers read and write text, they usually process it word by word, which uses a lot of memory and slows things down. The authors created a new method that groups parts of sentences into meaningful segments and compresses them, so the computer can work with shorter chunks. This method, called DSEI, helps large language models understand and generate text faster and using less memory. Tests showed it made the models almost half as confused about the text and more than twice as fast.
What this means in practice
- •For natural language processing engineers: Build faster and more memory-efficient language models by implementing segment-level latent inference in text generation applications.
- •For machine learning system architects: Design large-scale AI services that reduce hardware costs through compressed semantic representations during inference.
Tested on one dataset.
Authors
Peipei Li, Dongsen Zhang, Yuchen Liu, Wenjun Xu
Abstract
Large Language Models (LLMs) primarily perform inference at the token level, resulting in substantial memory overhead and compromised computational efficiency. In this paper, we propose a Dynamic Semantic Extraction and Inference (DSEI) framework, which achieves segment-level inference within the latent space through a two-stage training strategy. First, we construct a Dynamic Semantic Autoencoder (DSAE) via self-supervised learning. DSAE dynamically extracts segment-level semantics and compresses them into compact latent representations via adaptive semantic weighting and gated fusion. Subsequently, we integrate the DSAE into the LLM architecture and train the model to infer over dense latent space. DSEI substantially reduces both input and generation sequences and significantly enhances inference efficiency. Extensive experiments conducted on the Wanjuan dataset demonstrate that DSEI reduces perplexity by 48% compared to static sentence-level latent inference baseline. Furthermore, compared to standard LLMs using token-level inference, DSEI accelerates inference speed by 2.5$\times$ and reduces memory overhead by 90%.