Mamba with Hierarchical Memory: Solving Representation Bottleneck in Long Sequence Modeling
2026-08-03 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors found that a type of fast, efficient memory model called Mamba struggles with really long sequences because it has limited memory capacity. They created a new version, Hierarchical Memory Mamba (HMM), which mimics how humans use different memory levels by adding a working memory that summarizes paragraphs and a long-term memory that stores important info. This approach helps the model handle longer texts better and improves its ability to generalize across tasks. They showed that HMM performs better on tests involving retrieval and reasoning, with only a small increase in model size and training time.
Recurrent linear attentionMambaHierarchical memoryWorking memoryLong-term memoryParagraph-level semanticsSequence modelingCross-task generalizationPasskey RetrievalLongBench-E
Authors
Qinwen Wang, Jieping Luo, Aoxiang Qin, Ruoyu Zhao, Jianxiong Tang, Wei Zhang, Zhichao Lu, Luziwei Leng
Abstract
Recurrent linear attention models (RLAs) such as Mamba offer efficient linear-time sequence modeling as an alternative to Transformers, yet their fixed-capacity recurrent states limit long-sequence modeling. Drawing inspiration from hierarchical human memory, we propose Hierarchical Memory Mamba (HMM) to address this limitation. Building upon a pre-trained Mamba backbone, HMM integrates a lightweight working memory that extracts slow paragraph-level semantics (PLS) from the fast sensory memory embedded in the backbone's hidden states. The PLS is subsequently compressed into persistent long-term memory for task-relevant retrieval. The hierarchical processing of semantic information overcomes the representation bottleneck of RLAs and endows HMM cross-task generalization through parametric learning, which is not observed in other long-context enhanced Mamba variants. Evaluations on Passkey Retrieval and LongBench-E tasks demonstrate that HMM improves retrieval success by 34.3--37.1% and reasoning accuracy by 1.6--14.2% over strong Mamba-based models, while adding only 2% extra parameters and with minimal training overhead.