MixFormer: Linear Transformer with Mixture of Memory Experts
2026-08-10 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors introduce MixFormer, a new type of linear Transformer designed to better handle very long sequences of data. They address problems in existing models that struggle to remember important information over long stretches by using multiple memory experts and a special attention method that updates memory based on time and position. This approach helps the model focus on key past information and reduces the loss of memory details. Their tests on long text and image generation show MixFormer performs better and uses computing resources more efficiently.
State Space ModelsLinear TransformersMixture-of-ExpertsMemory CapacityLong-Range DependencyTime-Aware Linear AttentionExponential DecayPositional BiasSequence ModelingAttention Mechanism
Authors
Yu Guo, Lei Duan
Abstract
State Space Models (SSMs), as a mainstream research direction of linear Transformers, aim to achieve higher efficiency than standard Transformers in long-context modeling. However, existing SSMs suffer from limited input adaptivity and constrained memory capacity, leading to information loss when modeling ultra-long sequences. To address these limitations, we propose MixFormer, a novel linear Transformer that integrates a Mixture-of-Memory-Experts (MoE) mechanism. Specifically, the model maintains differentiated memory states through multiple collaborating memory experts and employs a novel Time-Aware Linear Attention (TALA) mechanism, which leverages learnable exponential decay functions and positional biases to dynamically update memory. This design enables the model to selectively reinforce important historical information while effectively mitigating memory dilution, substantially improving long-range dependency modeling. Experiments on long-sequence text and image generation tasks demonstrate that MixFormer not only achieves significant performance gains but also provides a more sustainable computational backbone for the next generation of web infrastructure.