Language modeling improves by using pure self-conditioning approach
Loopholing All the Way: Language Modeling with pure Self-Conditioning
Machine Learning
Summary
The paper looks at a technique called self-conditioning, which was previously a side trick in language models that predict text. The authors show it can be used as the main method to guide the model all the way through generating text-like information, without repeatedly converting between steps. They tested their approach, called LAW, on puzzles and text tasks and found it worked better or more accurately than previous methods. This suggests self-conditioning alone can be a powerful way to improve language generation models.
What this means in practice
- •For ai developers: Enhance language generation models with a pure self-conditioning technique for improved puzzle solving and text prediction accuracy.
- •For machine learning engineers: Implement self-conditioning based diffusion models to achieve more accurate outputs with fewer decoding operations in natural language processing tasks.
Authors
Adrien Deschenaux, Nadezhda Ilieva, Justin Deschenaux, Manuel Madeira, Dorina Thanou, Aurélien Lucchi, Antonio Orvieto, Caglar Gulcehre, Pascal Frossard
Abstract
Recent Diffusion Language Models (DLMs) use Self-Conditioning (SC), or Loopholing, to improve their performance. While SC is traditionally seen as an auxiliary heuristic, its empirical success raises the question: can a continuous generative model be based purely on Self-Conditioning? We answer affirmatively with LAW (``Loopholing All the Way''), which propagates the Expected Embedding under the denoiser's prediction across steps, decoding the tokens once, at the end. On Sudoku, LAW outperforms previous Discrete and Continuous Diffusion Models, including Loopholing Discrete Diffusion. On Maze, LAW is the most accurate from 2 to 64 NFE. After training on TinyGSM and sampling at temperature 1 with 1024 NFE, LAW solves 42.2% of GSM8K problems against 34.5% for uniform Loopholing Discrete Diffusion. Additionally, fine-tuning a pre-trained Uniform Diffusion Model (Duo) with LAW increases its accuracy from 17.2% to 48.1%. Finally, on OpenWebText, LAW pushes the Pareto frontier and approaches Autoregressive Models (ARMs) with 64 NFE. Find our code at https://github.com/ilievanadezhda/law.