Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?
2026-06-29 • Computation and Language
Computation and Language
AI summaryⓘ
The authors found that usual automatic ways to judge Chinese poetry don't work well, and asking people to evaluate is costly for lots of poems. They created Poller, a method using large language models (LLMs) that pretend to be the poem's author to judge the poem, mimicking human thought. Tests showed Poller closely matches human evaluations better than other automated methods, especially in judging poetry style and creativity. This approach makes it easier and more accurate to automatically evaluate poetry.
Chinese poetryautomatic evaluationhuman evaluationlarge language modelsPollerrhetorical techniquesdefamiliarizationpoetry interpretationliterary evaluation
Authors
Shanshan Wang, Derek F. Wong, Jingming Yao, Lidia S. Chao
Abstract
Traditional automatic evaluation methods have been shown to be unsuitable for modern Chinese poetry because of the distinct nature of this literary genre. Human evaluation remains reliable, but is expensive and not applicable to large-scale data. In this paper, we propose Poller (Poetry LLM Evaluator), a novel method leveraging large language models (LLMs) to evaluate the poetry understanding task. Specifically, our method requires LLMs to play the role of a poem's author with detailed information, thereby emulating human evaluation and judgment by adopting the poet's perspective. We conducted comprehensive experiments on multiple LLMs, evaluating the interpretations of poems across eight specialized dimensions. Experimental results demonstrate that our method effectively reduces the evaluation error between LLMs and humans. Especially for specific dimension evaluation, Poller-based LLMs achieve a 94.55% and 89.53% error reduction for rhetorical techniques and defamiliarization, respectively, compared to baseline methods. These performances are unattainable by conventional LLM evaluation methods. Experimental results from multiple LLMs across various dimensions validate the efficacy of our method. This work bridges the gap between automated efficiency and human expertise, establishing a foundation for automated evaluation in poetry-related tasks.