Summary
When large language models face questions about new information, they can include that information in three ways: in the text they process ('context'), by changing what they already know ('parameters'), or by adding special memory pieces ('representations'). This paper compares two methods—adding memory pieces (called Cartridges) and changing what the model already knows (fine-tuning)—to see which is better. The researchers found that memory pieces usually give more accurate answers and work better when dealing with many documents, but they can cause the model to forget old information more than other methods. Different ways of storing information have trade-offs in accuracy, cost, and what the model forgets.
What this means in practice
- •For machine learning engineers: Select document storage methods for updating language models to balance accuracy and memory use during deployment.
- •For information retrieval teams: Use memory-based document injection to improve answering accuracy in systems that retrieve multiple documents to handle knowledge updates.
Authors
Nathanaël Carraz Rakotonirina, Momchil Hardalov, Gonzalo Iglesias, Adrià de Gispert
Abstract
To answer questions outside of their pre-training data, large language models (LLMs) need access to new information, which can be presented in the context window as documents, encoded into the model's parameters, or injected as latent representations. However, each of these methods comes with different efficiency, cost, and performance trade-offs, with no single winner. We present a controlled comparison of representation-based (KV-cache based) and parametric (fine-tuning-based) adaptation methods on five knowledge-intensive benchmarks. We show that in the oracle setting, Cartridges (KV) are the most accurate injection method at nearly every storage budget, outperforming parametric methods by 10 points. Compaction (KV) matches Cartridges only at low compression rates, lagging behind the parametric methods by 10 points at rates higher than $50\times$. In the more realistic multi-document retrieval scenario, Cartridges are the only method that matches in-context learning (ICL), leading the parametric methods by 29 points and Compaction by 15 points. Nonetheless, Cartridges are also the only method, besides full fine-tuning and large MLP adapters, that suffers from catastrophic forgetting, i.e., a 6% performance degradation on control benchmarks, with 13% in coding.