Log-depth recurrent models offer scalable alternative for language tasks
Log-Depth Recurrent Language Modeling
Machine LearningComputation and Language
Summary
Language models help computers understand and predict text, but many rely on Transformers that use a lot of computing power, especially for longer sentences. The authors showed a new kind of model that processes text with less depth and can scale better while keeping good performance. This approach uses a balanced tree structure for predictions, making it faster without needing to look at the whole input at once. Their tests show it works well for longer texts and competes with common Transformer methods.
What this means in practice
- •For natural language processing engineers: Implement scalable language models that handle longer inputs with reduced computational depth and efficient runtime.
- •For speech recognition developers: Incorporate efficient autoregressive models for real-time transcription with strong performance on long sequences.
Authors
Yiqin Wang, Nuri Cingillioglu, Charles Pert
Abstract
Language modeling using Transformers has become commonplace despite their fixed computational depth and quadratic runtime with respect to input tokens. Recurrent models on the other hand offer linear depth but no parallel execution. In this work, we extend balanced-tree recursive operators from sequence encoding to autoregressive prediction, enabling all prefix representations to be computed with logarithmic depth and linear runtime. Our experiments provide an initial characterization of this model class, demonstrating robust length extrapolation and performance approaching that of ALiBi-based Transformers, highlighting its potential as an alternative architecture for language modeling.