Parallel language models speed up text generation with similar accuracy
Parallel Language Models via Variational Decoding
Machine LearningComputation and Language
Summary
Language models usually generate text one word at a time, which can be slow especially for bigger models. The authors propose a new approach called Parallel Language Models (PLMs) that generate many words at once by using hidden variables to keep the generated words consistent. They train these models using a special math trick that combines traditional training with new parallel decoding tasks. Their experiments show PLMs can generate text faster than older methods without losing quality and even outperform some on tasks like math or coding.
What this means in practice
- •For natural language processing engineers: Reduce inference time when deploying large language models in applications requiring fast text generation, such as chatbots or virtual assistants.
- •For software development teams: Improve speed and efficiency of coding assistant tools by integrating faster language models that maintain quality on programming benchmarks.
Authors
Felix Draxler, Robin C. Geyer, Daniel Geyfman, Farrin Marouf Sofian, Justus Will, Philipp Hildebrandt, Giosue Migliorini, Prakhar Srivastava, Gagan Sai Trinadh Duggaraju Peddiraju, Kushagra Pandey, Jorge da Silva Gonçalves, Elijah Tavares, Jan Niklas Groeneveld, Yang Meng, Julia E. Vogt, Stephan Mandt
Abstract
Autoregressive language models emit one token per forward pass. This causes high inference latency, in particular as models are scaled up for quality. In this paper, we introduce Parallel Language Models (PLMs), which generate mutually consistent tokens in one forward pass by routing dependencies through latent auxiliary variables. PLMs are trained using a variational bound that reduces to an autoregressive loss plus a parallel decoding loss. Empirically, PLMs are on a faster and better Pareto frontier than autoregressive and discrete diffusion baselines. Pretrained PLMs achieve near-autoregressive pretraining loss on model scales of 80M to 3B parameters, and speedup follows a power law in model size. When post-trained, our 1.3B model matches or exceeds an identically trained autoregressive model on math, coding, and instruction-following benchmarks, speeding up inference $2.3\text{-}5.7\times$. Together, we show that parallel generation can be trained from a single training objective at autoregressive quality. We will make our code available at https://github.com/mandt-lab/plm.