Human annotation boundaries stay stable labels shift over time
Boundaries Agree, Labels Do Not: Intra-Annotator Dynamics as a Kind of Training Data
Computation and Language
Summary
It's hard to say what is the correct way to label parts of a text because people can interpret it differently. The authors studied how one expert and some language models marked segments and their meanings in ancient myths at different times. They found that the expert consistently split the text in the same places but changed the labels they gave those segments, while the models behaved differently. This pattern can help check how reliable annotation data is and whether it looks human or machine-made.
What this means in practice
- •For data annotation teams: Identify and improve annotation data quality by detecting inconsistencies between boundary placement and label stability over time within the same annotator.
- •For language model developers: Evaluate training data contamination by comparing the temporal stability of human annotations to model outputs, improving model reliability.
Authors
Marharyta Shvets
Abstract
Data quality now matters as much as compute for training language models. Much training data comes from human annotation of text, and interpretive annotation has no ground truth that could settle what is "accurate". Two lines of work respond to this. One combines annotators into a "ground truth" and measures how well they agree with each other; the other treats their disagreement as a signal. Both compare different people at one point in time. We measure something else: how well one reader reproduces their own reading of the same text over time. One expert human reader and three LLM families segmented three Sumerian myths and labelled the causal function of each segment with one of seven states. Across runs months apart, the human cut the text in much the same places but named the segments differently, in every myth. The models show no such consistent pattern: their gap between the two layers is positive in some myths and negative in others, and its size varies. The human's label changes are not random: the runs go through much the same functions but start them one step apart, while model runs start them at the same places. We argue that this pattern is a usable measure of data quality and a contamination check: a "human" annotation whose labels are as stable as its boundaries, and whose functions start in sync, looks like a model's.