Smaller models learn literary sentiment from large language model scores
Distilling Literary Sentiment from LLM Annotations into an Ordinal BERT Model
Computation and Language
Summary
Human-labeled data to teach computers how to understand emotions in literature is hard to get. The authors looked at whether big AI models can label literary sentences for sentiment and then use those labels to train smaller models. They trained several smaller BERT models to guess sentiment scores from big AI-generated labels and tested them against human-labeled data. One smaller model even did better than the big AI teacher on sentences with clear emotions. They also found that just guessing neutral sentiment works well overall because many sentences are neutral, which can hide real improvements.
What this means in practice
- •For literary analysis software teams: Build smaller sentiment analysis tools for literature by training on large AI-generated labels, improving emotion detection in literary text.
- •For digital humanities developers: Enhance tools that interpret literary emotions by integrating models trained on AI-labeled sentence sentiment, supporting nuanced text understanding.
Tested on one dataset.
Authors
Artem Suslov, Takehiro Hashimoto
Abstract
The scarcity of human-annotated literary corpora limits the development of domain-specific sentiment analysis models. This study investigates whether LLM-generated annotations can supervise smaller models for literary sentiment analysis. We fine-tune five BERT models using ordinal regression on 10,889 literary sentences annotated by Gemma 4 31B on a ten-point sentiment scale. The models are evaluated against 1,923 human-annotated sentences from Hemingway's "The Old Man and the Sea", alongside the teacher model and conventional baselines. BERT-large achieves a Kendall's $τ$ of 0.589, approaching the teacher's 0.604. Although a constant-neutral predictor achieves the lowest overall mean absolute error (MAE), BERT-base outperforms the teacher on non-neutral sentences, achieving an MAE of 0.726 versus 1.116. These findings demonstrate the potential of LLM-generated supervision for developing smaller literary sentiment models, while highlighting the limitations of aggregate error metrics on predominantly neutral datasets.