Individual search histories help predict personal knowledge patterns
Individual Text Corpora Predict User-Specific Knowledge: Benchmarks of Individualized Knowledge Simulation
Information Retrieval
Summary
People leave clues about what they know in the words they search and read online. This study checked if large language models (LLMs) can guess people's knowledge using those clues. They found that one specific LLM could do better than people on some standard questions but struggled on new ones. Using the personal text collections to help the LLM guess individual answers worked better than random guessing, but the predictions were not very confident or perfectly accurate. The authors also created ways to measure how well these guesses match actual knowledge.
What this means in practice
- •For personalized learning platform developers: Create tools that adapt quizzes or content recommendations based on users’ individual knowledge predicted from their search histories.
- •For digital assistants teams: Improve response accuracy by integrating user-specific text data to better model individual knowledge and preferences.
Authors
Christoph Wigbels, Ali Abusaleh, Markus T. Jansen, Alexander Mehler, Manuel Schaaf, Markus J. Hofmann
Abstract
This study examines whether individual text corpora (ICs) from search histories can be used to simulate individual knowledge. We collected ICs from 316 adults, who answered 36 multiple-choice knowledge items, and compared several large language models (LLMs) on this task, of which only Qwen3-1.7B proved viable. After task-specific fine-tuning via Low-Rank Adaptation (LoRA), Qwen3-1.7B outperformed both participants and a representative German norm sample on publicly available items. On non-public questions, however, the LLM performed worse than our participants, suggesting possible training data contamination for the public questions. When integrating ICs into retrieval-augmented generation to predict individual responses, LLM-participant Match accuracies significantly exceeded chance, which demonstrates a detectable individual knowledge signal. The probabilities assigned to the participants' answers were, however, low and far below the probability of correct answers, indicating poor calibration toward individual response patterns. Knowledge-gap prediction was sub-optimal, though it improved for corpora exceeding five million tokens. We discuss our entropy based evaluation benchmarks as calibration indices for individualized knowledge simulation.