Summary
This paper looks at whether machines and humans agree on identifying certain storytelling features in Turkish narratives. The authors checked if a rule-based system and several large language models (LLMs) could reliably label scenes with specific narrative techniques compared to human judges. They found that overall agreement was low, especially for a key feature called materialized metaphor, where machine and human labels often didn’t match. The discrepancy could mean that the feature is too inferential for current automated methods or that the feature’s definition is too unclear for consistent human or machine labeling. The authors did not have a second human check to decide which explanation is more likely.
What this means in practice
- •For natural language processing engineers: Use this evaluation to understand limits of automatic narrative feature labeling and improve annotation tools for Turkish storytelling datasets.
- •For computational linguistics teams: Assess reliability challenges when combining rule-based and LLM annotations for complex, inferential narrative elements in under-resourced languages.
Tested on one dataset.
Abstract
Datasets that ship automatically generated feature annotations invite a question rarely asked of them: would a human agree with those labels? This report answers that for the Objective Projection corpus, a Turkish narrative dataset whose scenes carry a per-scene applied_rules field from a rule-based detector over six craft features -- two prohibitions (emotion labelling, simile) and four positive techniques (materialized metaphor, micro-focus, temporal anchor, atmosphere contradiction). Three studies are reported. Study 1 ($n = 120$) scores the detector against blind labels from the scheme's own author. Study 2 ($n = 100$, a disjoint scene set) scores the detector plus Gemini 2.5 Flash and Grok against an independent non-expert rater whose labels were locked before any machine ran. Study 2b re-runs the identical protocol with Claude Fable 5 (High) and ChatGPT 5.5. The central result concerns one rule. On materialized metaphor -- closest to the methodology's theoretical core -- the five machine labellers returned positive rates of $0$, $1$, $40$, $72$ and $78$ out of $100$ scenes, against a human count of $9$. Cohen's $κ$ was at or indistinguishable from chance for five of six labellers, across both human references and both scene sets: $0.004$, $0.015$, $0.000$, $0.019$, $0.027$. Raw agreement ranged from $74.7\%$ to $84.5\%$, an artefact of class imbalance rather than a sign of competence. We deliberately do not resolve this into a single story. Two readings survive: the feature is genuinely inferential and beyond current automatic detection, or the rule's definition is not yet operational enough for any rater to apply consistently -- including the human. Distinguishing them needs a second independent human rater, which this report does not have and therefore does not claim.