Small language models personalized to user text improve memory simulation

From Retrieval to Weights: Parametric Individualization of Small Language Models with Individual Text Corpora

Computation and LanguageInformation Retrieval

Summary

This work studies how small language models can be personalized by learning from a person’s own text documents, kind of like how our brain remembers facts from our experiences. The authors collected web search histories from hundreds of people answering trivia questions and trained small models to include each person’s text into the model’s internal memory. These personalized models could better predict that same person’s writing, showing the model truly learned from individual data. However, when tested on general knowledge questions, the models added overall knowledge but didn't perfectly match the person’s own knowledge style. This approach helps explore how to build AI that simulates personal memory and could lead to individual tutoring systems.

What this means in practice

  • For personalized tutoring developers: Create AI tutors that adapt their knowledge base specifically to an individual’s own text and learning history for more tailored teaching.$Commercial implications: Enables personalized AI tutoring systems that can incorporate a student’s own materials directly into a compact language model.
  • For chatbot designers: Build chatbots that better simulate an individual user’s voice and knowledge by consolidating their text into the model’s parameters.

Authors

Christoph Wigbels, Ali Abusaleh, Markus T. Jansen, Alexander Mehler, Markus J. Hofmann

Abstract

We approach a cognitive simulation perspective on episodic and semantic memory in multiple-choice question answering by incorporating text from individual text corpora (ITC) into retrieval-augmented generation and DoRA fine-tuning. We web-crawl the search histories of 515 participants who answered 36 multiple-choice knowledge items and analyze a stratified subsample of 150 participants. For each participant, one DoRA adapter consolidates their ITC into a small language model (SLM) whose baseline correctness falls below the participants' lowest quartile. The adapter measurably writes the ITC into the weights: it fits its own participant's held-out text better than other participants' texts (dz =1.27), an individuality effect that increases with ITC size in rank order. On the generalized knowledge test, however, the adapter adds knowledge rather than alignment with the individual: log-loss match improves, whereas match accuracy under a bias-corrected PMI readout does not, and retrieval adds nothing on top. Our results demonstrate that ITCs can be consolidated into the weights of SLMs, an encouraging basis for individualized tutoring agents, and we discuss how to move from there toward a realistic simulation of episodic and semantic memory at the individual level.
View PDFOpen arXiv