Summary
This study explored whether the text someone generates while searching online can reflect what they know. The researchers used datasets from hundreds of adults along with large language AI models to see if these models could guess answers people would give on quizzes. One AI model, after special fine-tuning, sometimes did better than people on known questions but struggled on new ones, suggesting it might have seen some quiz data before. By adding the individuals’ search texts into the AI’s process, the model’s guesses matched people’s answers better than random chance, showing personal knowledge clues exist in these texts. However, the AI wasn’t very good at estimating confidence or identifying gaps in knowledge, especially with smaller amounts of text.
individual text corporasearch historylarge language modelsknowledge simulationfine-tuningLow-Rank Adaptation (LoRA)retrieval-augmented generationmodel calibrationknowledge-gap predictionentropy evaluation
Authors
Christoph Wigbels, Ali Abusaleh, Markus T. Jansen, Alexander Mehler, Manuel Schaaf, Markus J. Hofmann
Abstract
This study examines whether individual text corpora (ICs) from search histories can be used to simulate individual knowledge. We collected ICs from 316 adults, who answered 36 multiple-choice knowledge items, and compared several large language models (LLMs) on this task, of which only Qwen3-1.7B proved viable. After task-specific fine-tuning via Low-Rank Adaptation (LoRA), Qwen3-1.7B outperformed both participants and a representative German norm sample on publicly available items. On non-public questions, however, the LLM performed worse than our participants, suggesting possible training data contamination for the public questions. When integrating ICs into retrieval-augmented generation to predict individual responses, LLM-participant Match accuracies significantly exceeded chance, which demonstrates a detectable individual knowledge signal. The probabilities assigned to the participants' answers were, however, low and far below the probability of correct answers, indicating poor calibration toward individual response patterns. Knowledge-gap prediction was sub-optimal, though it improved for corpora exceeding five million tokens. We discuss our entropy based evaluation benchmarks as calibration indices for individualized knowledge simulation.