Cortex improves long term memory in large language models
Cortex: Consolidating Context into Parametric Memory
Computation and LanguageArtificial IntelligenceComputer Vision and Pattern RecognitionMachine LearningMultiagent Systems
Summary
Large language models get better when given extra context, but they often lose those improvements once the context is gone. The authors developed a method called Cortex that helps models keep the benefits of context by 'internalizing' what they learned, much like how memories consolidate in the brain. Cortex works by creating smart questions to anticipate how the model will use the context and then updating the model in a way that remembers the useful info without losing other knowledge. This helps models keep improvements even after many updates.
What this means in practice
- •For ai model engineers: Maintain improvements from contextual information in language models while preserving existing capabilities during iterative model updates.
- •For multimodal ai developers: Integrate Cortex to consolidate context effects efficiently in text and multimodal models, improving retention after sequential updates.
Authors
Jintian Zhang, Feng He, Xinnian Liang, Ziwen Xu, Haoming Xu, Haojing Huang, Shuangzhi Wu, Ningyu Zhang
Abstract
Context can improve large language model performance, but these gains may be lost after context removal. Inspired by biological systems consolidation, we formulate Memory Consolidation (MC) as internalizing these gains while preserving non-target capabilities, connecting context-specified editing, skill learning, and behavioral steering under a common objective. We introduce a five-task-family testbed measuring consolidation and retention under single, batch, and sequential updates. We propose Cortex, which generates prospective queries to anticipate uses of context and distills context-conditioned outputs into mergeable low-rank or structural feed-forward network updates. Across 4B--120B dense and mixture-of-experts backbones, including both text-only and multimodal models, Cortex lies on the empirical consolidation--retention Pareto frontier in most evaluated settings. After 50 sequential factual edits on a 14B model, Cortex-Structural recovers 88.9% of the context-induced gain while retaining 99.8% of original non-target performance.