Detecting and Guiding LLM-Generated Korean Poetry with Interpretable Form-level Features
Computation and Language
Summary
The authors study how large language models (LLMs) struggle to write Korean poetry that looks like real human poems. They identify four key differences between human and AI poems, such as poem length and rhythm, and use these to create five features that help detect if a poem is written by a human or an AI. Their detection method works better than previous ones, even on unseen AI models. They also guide AI models to produce poems that better match human style, confirmed by expert review and statistical analysis. This shows that using clear, language-specific features helps both spotting and creating more human-like poetry with AI.
Authors
Keunhyeung Park, Seunguk Yu, YoungBin Kim
Abstract
LLMs often struggle with modern Korean poetry, producing outputs that resemble "line-broken prose." We address two coupled tasks: detecting whether a Korean poem is human- or LLM-authored, and guiding LLMs to generate poetry closer in form to human writing. We quantify the human-LLM gap along four form-level linguistic dimensions: output length (Volume), the diversity and connective use of line-final forms (Structure Variation), the irregularity of line lengths (Rhythmic Irregularity), and adherence to standard orthography (Normative Adherence). We operationalize these dimensions as five interpretable features. For detection, a logistic regression classifier over these five features attains an average AUC-ROC of 83.60 in zero-shot out-of-distribution detection across seven unseen LLMs, versus 75.84 for the strongest baseline in our comparison, KatFishNet, an absolute gain of 7.76 AUC points and a 10.23% relative improvement; one generator-specific punctuation pattern outside our taxonomy remains a boundary case. For generation, expert evaluation on GPT-5.2 prefers feature-guided poems over the unconstrained baseline, and analyses across GPT-5.2 and Gemini-3 show that targeted length, rhythm, and ending statistics move toward the human distribution. These results suggest that interpretable, language-specific features can bridge the diagnosis and guidance of LLM-generated poetry.