SpectralShift improves long context handling in gated DeltaNet models
SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization
Computation and Language
Summary
Many AI models struggle to remember and use information from very long texts. The authors studied a type of AI layer called Gated DeltaNet and found that certain patterns in how information fades over time help it remember longer contexts better. They created a method named SpectralShift to adjust these patterns during training, making the model better at handling long texts. Their tests show that this method consistently helps the model learn to use more information from longer passages more effectively.
What this means in practice
- •For natural language processing engineers: Extend and improve AI models’ ability to process very long documents by using SpectralShift during model retraining.
- •For software developers of ai assistants: Enhance virtual assistants’ understanding of long conversations or documents by incorporating SpectralShift-enhanced models.
Authors
Zian Liu, Yiwen Hu, Zican Dong, Tian Xie, Wayne Xin Zhao, Yucheng Ding, Ran Tao, Bryan Dai
Abstract
Recently, linear attention layers have been increasingly adopted to replace softmax attention at scale for long-context modeling. However, existing context extension approaches typically apply continued pretraining directly without modifying these layers, overlooking the spectral properties of linear attention state dynamics. In this work, we study long-context extension of Gated DeltaNet (GDN) from a spectral perspective of transition matrix and identify two essential factors governing long-range information retrieval: (1) a sufficiently broad slow spectral band aligned with the target dependency length, and (2) the preservation of fast-decaying modes for state clearing and context switching. Based on this observation, we propose SpectralShift, a spectral reparameterization approach for long-context continual pretraining of GDNs. Specifically, SpectralShift reparameterizes the alpha projections initialization to reshape the decay spectrum by enhancing slow propagation capacity, and further introduces a learning-rate scaling for alpha projections to facilitate long-context training. Experiments show that SpectralShift consistently improves long-context capabilities over training, providing an effective and efficient solution for extending context windows of linear attention models. The code has been open-sourced at https://github.com/RUCAIBox/GDN-SpectralShift.