Language models evaluated for explaining cultural references with evidence
From annotation to reasoning: Culture in language models
Computation and Language
Summary
Evaluating language models is tricky when multiple interpretations can all be correct, especially in culture and literature. The authors suggest literary interpretation helps study how models explain and support cultural references in texts rather than just agreeing on facts. They propose new ways to evaluate models that include evidence and allow for different valid opinions, starting with Danish literature to improve AI’s cultural understanding. This work looks beyond simple test scores toward deeper reasoning about cultural meaning.
What this means in practice
- •For natural language processing teams: Develop evaluation methods for language models that assess cultural understanding by checking how well models support interpretations with evidence.
- •For digital humanities developers: Create tools processing literary texts that link cultural references to historical contexts allowing nuanced interpretation and scholarly disagreement.
A position paper. It proposes an approach and reports no results.
Authors
Daniel Hershcovich, Alexander Conroy, Jens Bjerring-Hansen
Abstract
How should we evaluate language models when more than one interpretation can be right? Cultural benchmarks often test factual knowledge, agreement with survey responses, or recognition of a predefined meaning. These tasks leave open whether a model can explain how a cultural reference works in a particular text, support a reading with evidence, or revise it after criticism. This is a question of interpretive depth, complementary to the breadth of cultural coverage. We argue that literary interpretation offers a useful setting for studying these capabilities. We focus on cultural referencing and reuse: how texts invoke, repeat, and transform earlier expressions across historical and linguistic contexts. Our central claim is that literary scholars can disagree about an interpretation while recognizing the quality of its support. We propose linking evidence-centered benchmarks, evaluation that preserves scholarly disagreement, and model-development experiments on literary data, contextual resources, and scholarly feedback. Danish literature provides a concrete starting point, with implications for other languages and domains. The aim is to develop alternative evaluation strategies that go beyond conventional benchmark metrics and guide model development toward cultural robustness in AI systems.