Context updates improve large language model performance on evolving tasks
Towards Evolving Context Parameterization for Large Language Models
Computation and LanguageArtificial Intelligence
Summary
Large language models (LLMs) often reuse past information to answer questions more efficiently, but they struggle when that information changes over time. The authors introduced a new task, called MUSE, to test how well models update what they know while keeping unchanged info intact. They proposed a method called PLUME that combines old and new information dynamically without retraining the model. Tests showed PLUME helps LLMs handle changing data much better than before.
What this means in practice
- •For chatbot developers: Update chatbot knowledge continuously while preserving prior context for more accurate conversations.
- •For customer support teams: Maintain up-to-date automated responses that reflect recent policy changes without losing previous relevant information.
Authors
Xiaobing Shi, Zherui Li, Yiming Jiang, Kun Wang, Yufei Guo
Abstract
Context parameterization enables large language models (LLMs) to internalize contexts into reusable model parameters, avoiding repeated processing across subsequent queries. However, existing methods typically assume static contexts and lack explicit mechanisms for distinguishing validity states under continual updates. To study this real-world scenario, we formalized the Memory Updating with Sequential Evolution (MUSE) task and constructed MUSE-bench to evaluate update incorporation and unaffected-information preservation. The resulting challenge requires preserving the global state while adjusting the contribution of memory evidence. Motivated by this, we proposed PLUME, a training-free method that constructs a global update representation, activates memory evidence to form a local parameter view, and adaptively integrates their predictions during decoding. Comprehensive evaluation on MUSE-bench demonstrated PLUME's effectiveness in sequential evolution settings, yielding relative improvements of 29.9% in average ROUGE-L Recall and 54.9% in LLM-as-a-Judge. Our codes are available at: https://github.com/xiaobingshi-LLM/PLUME.