AI summaryⓘ
The authors studied how to use machine learning to automatically label parts of Voltaire's works with themes, a task usually done by hand and quite time-consuming. They treated this as a problem where a model assigns multiple relevant index labels to each page of text. By testing various models, including big language models fine-tuned with a special method called LoRA, they found their best model reached a reasonable accuracy score, noting that some disagreements with human labels can still be meaningful. They also examined how well their models generalize across different texts and looked closely at specific text features that are hard for machines to label. Overall, the authors' work offers insights for making thematic indexing easier in large collections of literary and historical documents.
thematic indexingmulti-label classificationmachine learninglarge language models (LLMs)Low-Rank Adaptation (LoRA)Mistral modelsF1 scorecross-corpus generalisationliterary analysistext annotation
Authors
Miguel Arana-Catania, Gillian Pink, Glenn Roe
Abstract
Thematic indexing -- the practice of assigning structured conceptual labels to sections of text -- is essential to scholarly access in large-scale literary and historical editions, yet it remains a largely manual, labour-intensive process. This paper explores the application of machine learning to automatic thematic indexing, using two substantial sub-corpora of the Complete Works of Voltaire as a test case: the Essai sur les mœurs et l'esprit des nations and the Questions sur l'Encyclopédie. The task is framed as a multi-label classification problem, in which a model must assign the set of index entries that a professional indexer would apply to a given page of text. We compare a range of approaches -- from encoder-based models with classification heads to generative large language models (LLMs) fine-tuned via Low-Rank Adaptation (LoRA) -- spanning model sizes from approximately 3 to 120 billion parameters. Our best-performing model, from the Mistral family in a 4-bit quantised configuration, achieves F1 scores of up to 0.67; we argue that these figures represent lower bounds, given the inherent subjectivity of professional indexing and the frequency with which model predictions prove semantically valid despite diverging from the print index. We further evaluate cross-corpus generalisation and conduct a detailed qualitative analysis of model behaviour on literary and rhetorical features of the source texts that prove particularly resistant to automated treatment. Our findings have implications for the broader challenge of providing structured thematic access to large-scale literary and historical corpora.