WhoSaidIt: Human-LLM Collaborative Annotation for Text-Based Multilingual Speaker-Attribute Classification
2026-05-25 • Computation and Language
Computation and Language
AI summaryⓘ
The authors address the challenge of labeling speaker attributes from text, which can be unclear and varies across languages and cultures. They created a process where humans and large language models (LLMs) work together to improve and stabilize these labels by focusing on disagreements and expert input. Using this method, they built a new multilingual dataset called WhoSaidIt with nine speaker attributes. Their analysis shows notable differences in how attributes are annotated across languages and highlights both what LLMs do well and where they struggle in this task.
speaker attributesannotationmultilinguallarge language modelsre-annotationdatasetcross-lingual differencesdisagreement samplingrationalemodel benchmarking
Authors
Lingyu Gao, Will Monroe, David Smith, Meghan Jemison, Jackie Lee
Abstract
Annotating speaker attributes from text is inherently ambiguous, particularly in multilingual settings where demographic and social cues are implicit and culturally variable. We propose a human-large language model (LLM) collaborative re-annotation framework for stabilizing multilingual speaker-attribute labels under practical resource constraints. Starting from a noisy corpus, we use LLMs to surface recurring annotation rationales through iterative interaction with experts, and apply disagreement-focused sampling for targeted re-annotation. Using this framework, we construct WhoSaidIt, a multilingual dataset covering nine speaker-attribute labels. We quantify divergence between original and revised annotations, benchmark recent LLMs, and analyze the effect of explicit rationales on model behavior. Our results reveal substantial cross-lingual differences in annotation decisions and demonstrate both the strengths and limitations of LLMs in speaker-attribute classification.