Large language models improve knowledge use with efficient chunk memory

Deep Fusion Memory: A Chunk-Level Parametric Memory for Large Language Models

Computation and Language

Summary

Large language models often struggle to remember or use facts not covered in their initial training. The authors propose Deep Fusion Memory (DFM), a method that predicts useful information from chunks of text to help the model remember better without slowing it down. This method merges predicted knowledge directly into the model's workings instead of just combining outputs, making it faster and requiring less data to train. Tests on question answering tasks and domain adaptation show this approach improves model performance significantly.

What this means in practice

  • For nlp engineers: Enhance question answering systems with faster and more accurate external knowledge integration without online retrieval overhead.
  • For software developers: Adapt large language models more efficiently to specialized domains using chunk-level memory fusion for improved task performance.

Authors

Rubin Wei, Siyuan Sun, Hao Sun, Yuliang Liu, Xiang Shi, Jiarui Wang, Junming Zhang, Zhouhan Lin

Abstract

Large language models need effective ways to acquire and use external knowledge beyond what they retain from pretraining. Continued pretraining risks catastrophic forgetting, while retrieval at inference time can add substantial overhead. External parametric memory avoids online retrieval, but constructing its training data requires extensive offline search, and output-level interpolation provides only shallow fusion. We introduce Deep Fusion Memory (DFM), a parametric memory architecture that reduces this construction workload and incorporates learned memory into intermediate hidden states. A causal memory predictor learns to generate memory from the preceding context, using retrieved chunk embeddings from the corpus as supervision. The predicted memory is shared across tokens within each chunk and fused into their hidden states through gated cross-attention. Memory prediction and fusion are trained end to end with the backbone frozen, and inference requires no explicit retrieval. Compared with existing parametric memory approaches, which often construct token-level memory targets, DFM requires $32\times$ fewer retrieval queries and $2048\times$ fewer retrieved items to construct its training data. Across six QA tasks, DFM performs better over the strongest baseline by 12.71% on Mistral-7B-v0.3 and 11.19% on Qwen3-8B-Base. Beyond QA, domain adaptation yields average gains of 6.51 and 4.38 points over Qwen3-4B-Base and Qwen3-8B-Base, respectively.