Activation and parametric memory work together in few-shot learning
Complementary Roles of Activation and Parametric Memory in Few-Shot Learning
Computation and Language
Summary
Large language models use two kinds of memory: activation memory, which keeps recent information active, and parametric memory, which changes with training. The authors explore how these memories work in few-shot learning, where models learn tasks from only a few examples. They found that activation memory is better at recalling facts, but neither activation nor parametric memory alone is enough to solve certain combined tasks. Instead, both memories must work together, activating different neurons to handle complex problems.
What this means in practice
- •For machine learning engineers: Design few-shot learning systems that integrate activation and parametric memories for complex task performance.
- •For ai system architects: Improve large language model architectures by optimizing how activation and parametric memories collaborate at the neuron level.
Authors
Miaohe Niu, Runsong Zhao, Xinyu Liu, Bo Jin, Yucheng Qiao, Chunliang Zhang, Jingbo Zhu, Tong Xiao
Abstract
At test time, large language models (LLMs) can encode historical information in activation memory (i.e., KV caches) and parametric memory (i.e., updated parameters). While activation memory is generally considered effective for factual recall and parametric memory for learning new tasks, their interplay remains unclear. In this work, we systematically investigate the role of memory in few-shot learning through controlled experiments. We find that activation memory is superior for recalling facts, whereas parametric memory does not consistently outperform activation memory in task learning. Moreover, our experiments show that the composite task, Conditional Arithmetic, requires the synergy of both memory types. Through neuron-level analysis, we find that the model activates distinct sets of neurons when accessing the same historical information through activation versus parametric memory. When both memory types are combined, the model recruits neurons from both sets, which is crucial for solving Conditional Arithmetic. These findings suggest that neither memory mechanism alone is sufficient for this composite task, highlighting the importance of their collaboration.